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Chapter  1 
INTRODUCTION 

As  stated  in  Air  Force  Manual  1-1,  the  Air  Force's 
role  in  national  defense  is  to  organize,  train,  and  equip 
its  forces  for  conduct  of  prompt  and  sustained  aerospace 
operations  essential  to  the  effective  prosecution  of  war 
(27:2-3).  A shortage  of  available  weapon  systems  would 
severely  hamper  the  Air  Force's  ability  to  perform  its  de- 
signated national  defense  role  and  decrease  our  nation's 
military  combat  readiness  (17s 25). 

In  1976,  Gansler,  then  Deputy  Assistant  Secretary 
of  Defense  for  Material  Acquisition,  stated  that  the  De- 
partment of  Defense  (DOD)  has  recognized  the  need  for  know- 
ing what  parameters  should  be  designed  into  new  weapon  sys- 
tems to  increase  their  availability  while  decreasing  their 
logistic  support  requirements  (10:1).  He  emphasized  that 
the  DOD  has  given  considerable  attention  to  improving 
availability  by  designing  more  reliable  and  maintainable 
equipment  into  its  new  weapon  systems.  DOD  is  concerned 
when  inadequate  equipment  reliability  and  poor  maintain- 
ability design  cause  aircraft  to  be  grounded,  to  fly  in  a 
degraded  condition,  or  to  be  unavailable  for  the  full  combat 
mission  for  which  procured  (10:1). 
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Efforts  have  been  made  by  the  Air  Force  to  foresee 
and  design  for  problems  that  would  be  encountered  during  the 
operational  life  of  its  weapon  systems.  Reliability  and 
maintainability  have  been  and  continue  to  be  highlighted 
during  the  design  of  newer  Air  Force  weapon  systems.  The 
F-15  aircraft,  for  example,  has  been  designed  with  a high 
reliability  parts  program  and  a stringent  test  effort  to 
provide  maximum  equipment  reliability  and  maintainability 
(16:40).  As  Gansler  stressed,  Air  Force  test  programs 
should  incorporate  the  flexibility  to  permit  attainment  of  a 
proper  balance  of  logistic  support  requirements  and  per- 
formance requirements.  Hardware  must  be  designed  to  em- 
phasize simple,  rugged,  and  easily  maintained  designs. 
However,  as  Gansler  further  emphasized,  such  an  approach 
requires  funds  and  schedules  for  design  flexibility.  Man- 
agers, both  in  the  Government  and  in  private  industry, 
should  plan  their  programs  to  accommodate  these  requirements 
so  as  to  achieve  low-cost,  simple,  highly  reliable  and  main- 
tainable weapon  system  designs  (10:1). 

Imperfect  foresight  and  use  of  systems  beyond  their 
initially  expected  life  have  contributed  to  deficiencies 
being  identified  after  the  system  has  become  operational 
(11:1).  While  every  attempt  is  made  to  initially  design 
equipment  reliability  and  maintainability  into  the  weapon 
system  equipment,  deficiencies  are  often  not  discovered  un- 
til after  the  weapon  system  has  entered  the  inventory  and 
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production  has  ceased.  Changes  made  to  correct  these  de- 
ficiencies are  called  modifications  (11:1). 

MODIFICATIONS 

Only  recently  has  the  Air  Force  emphasized  that 
reliability  be  designed  into  new  weapon  systems.  Former 
Deputy  Secretary  of  Defense  Clements  on  February  28,  1976, 
issued  a memorandum  to  the  Secretaries  of  the  Military  De- 
partments that  tasked  them  to  develop  weapons  that  cost 
less  to  operate  and  support  by  designing  in  greater  equip- 
ment reliability  and  maintainability  (22:1).  Another  sig- 
nificant area  for  reducing  O&S  costs  is  in  existing  Air 
Force  in-inventory  aircraft.  The  Air  Force  has  many  aging 
weapon  systems  in  its  inventory.  For  example,  the  F-4  air- 
craft entered  the  Air  Force  inventory  in  FY  63  and  the  F-lll 
aircraft  in  FY  66  (6:3). 

Although  the  Air  Force  is  faced  with  aging  weapon 
systems  which  possess  equipment  that  lack  the  current 
"state-of-the-art"  improvements,  budgetary  constraints  have 
restricted  the  funds  available  for  accomplishment  of  modi- 
fications. Therefore,  as  Shrontz,  former  Assistant  Secre- 
tary of  Defense  for  Installation  and  Logistics,  stated: 

. . . approval  of  modifications  has  become 
more  stringently  controlled  so  that  the  greatest 
benefits  possible  in  terms  of  improved  reliabil- 
ity and  maintainability  can  be  obtained  [22: l]. 

He  emphasized  that  the  Department  of  Defense  must  be  highly 
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critical  of  approving  a specific  modification  and  must  in- 
sure that  the  modification  will  produce  the  maximum  improve- 
ment  to  aircraft  availability,  i.e.,  improve  equipment 

reliability  and  reduce  manhours  required  for  support.  i 

Modification  of  in-use  systems  is  a very  expensive 
proposition.  Indeed,  the  Deputy  Chief  of  Naval  Material 
for  Reliability  and  Maintainability  has  stated  that 
".  . . field  retrofit  to  correct  poor  design  is  the  most 
expensive  and  disruptive  approach  to  reliability  improve- 
ment  [34:16].”  Since  modifications  are  so  costly  and  crit- 
ical to  systems  improvement,  the  Air  Force  published  Air 
Force  Regulation  (AFR)  57-4  which  provides  guidance  for 
justifying,  submitting,  approving,  and  accomplishing  modi- 
fications. There  are  several  classes  of  modifications.  The 
basis  of  classification  is  the  nature  of  the  change  or  the 
degree  of  improvement  (28:10-11).  Each  of  the  classes  can 
be  defined  as  follows: 

Class  I.  A temporary  change  to  accomplish  a spe- 
cial mission  for  a special  purpose. 

Class  II.  A temporary  means  to  support  research 
and  development  or  operational  test  and  evaluation  programs. 

Class  III.  Required  to  attain  production 
continuity . 

Class  IV.  Required  to  correct  an  observed  material 
deficiency;  this  class  of  modification  is  divided  into 
three  subclasses: 
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1.  Class  IVA.  Required  to  insure  safety  of 
personnel,  systems,  or  equipment  by  eliminating 
operational,  nuclear,  or  physical  hazard  and  if 
uncorrected,  the  hazard  would  ground  the  system 
or  equipment,  restrict  flight  or  ground  opera- 
tions, or  result  in  unacceptable  risk  to 
personnel  [28:10]. 

2.  Class  IVB.  Required  to  correct  a deficiency 
including  one  that  affects  reliability  and  main- 
tainability and  if  uncorrected  the  deficiency 
would  cause  mission  failures,  impede  the  system 
or  equipment  mission  accomplishment  of  other  sys- 
tems or  equipment  within  the  defense  or  civilian 
community  [28:10]. 

3.  Class  IVC.  Required  for  logistic  support 

purposes  and  one  of  the  following  benefits  will 
result:  significant  improvement  in  maintain- 

ability or  service  life;  improved  logistic  sup- 
port by  modification  of  present  equipment  in 
lieu  of  procurement;  reduce  costs  by  standardiz- 
ing equipment  configuration  [28:11]. 

Class  V.  Required  to  satisfy  a requirement  for  a 

new  capability  which  constitutes  a change  in  mission. 


CLASS  IV  MODIFICATION  PROCESS 


The  Class  IV  modification  process  is  generally  ini- 
tiated by  a reported  deficiency  from  an  operating  unit. 

The  report  of  a deficiency  may  take  several  forms.  The 
first  of  these  are  the  Incident  Report  and  Emergency  Un- 
satisfactory Material  Report  which  are  submitted  by  the 
operating  unit  directly  to  the  responsible  Air  Logistics 
Center  (ALC).  The  second  is  the  Modification  Proposal  which 
is  submitted  through  channels  to  the  responsible  Air 
Logistic  Center  (6:26). 
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In  a briefing  at  a 1977  conference  at  Wright- 
Patterson  AFB  (29:7),  it  was  emphasized  that  the  respon- 
sible Air  Logistics  Center  System  Management  or  Item  Man- 
agement Division  must  evaluate  the  reported  deficiency 
through  establishment  of  a Material  Improvement  Project. 
After  evaluation  in  the  responsible  System  or  Item  Manage- 
ment Division,  the  Material  Improvement  Project  is  for- 
warded to  the  Service  Engineering  Division  for  an  engineer- 
ing analysis.  If  a modification  is  the  only  means  by  which 
the  deficiency  can  be  eliminated,  the  Service  Engineering 
Division  develops  the  complete  engineering  solution. 

When  the  deficiency  is  initially  reported  by  a 
command  modification  proposal,  the  modification  proposal 
is  reviewed  in  detail  by  base-level  and  major  command 
logistics  managers  prior  to  its  being  forwarded  to  the  Air 
Logistics  Center  for  approval  (28:13). 

Once  the  engineering  solution  to  the  deficiency  has 
been  determined,  the  System  Manager  or  Item  Manager  obtains 
coordination  or  certification  from  the  using  commands  and 
other  involved  agencies.  The  certification  process  in- 
volves Configuration  Control  Board  (CCB)  review  of  the 
engineering  solution  (6:33). 

AFR  57-^  requires  that  each  major  command  establish 
a CCB  which  is  responsible  for  reviewing  all  modification 
proposals  originating  in  or  which  affect  the  command 
(28:4-7).  The  review  must  provide  certification  that  the 
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modification  is  required  and  must  include  a statement  of 
the  benefits  and  impact  for  approval  of  the  modification. 

There  are  also  CCBs  at  the  Air  Logistic  Centers, 

Air  Force  Logistic  Command  (AFLC)  Headquarters,  and  at  the 
Air  Staff  (6:33).  The  ALC  boards  consider  all  proposed 
modifications  that  have  a total  cost  of  less  than  $500,000 
(33).  If  the  total  cost  exceeds  $500,000  and  the  ALC 
recommends  approval,  the  proposed  modification  is  forwarded 
to  the  AFLC  board  for  approval.  However,  if  the  total  cost 
of  the  proposed  modification  exceeds  $5  million  then  the 
Air  Staff  must  approve  it.  Each  board  tries  to  obtain  the 
maximum  overall  benefit  for  the  Air  Force.  The  AFLC  board, 
however,  decides  how  the  majority  of  the  Air  Force  modi- 
fication dollars  will  be  spent  (33) • 

The  review  of  a modification  proposal  by  a CCB  is 
accomplished  by  reviewing  the  AFLC  Form  48  (Configuration 
Control  Board  Item  Record)  as  discussed  by  Baker  et  al.  in 
their  study  (2:26).  This  form  is  completed  by  the  System 
Manager  and  describes  the  technical  and  cost  characteris- 
tics of  the  proposed  modification.  The  AFLC  Form  48  is 
the  major  instrument  which  the  CCBs  use  to  evaluate  the 
proposed  benefits  of  a modification  (2:26). 

STATEMENT  OF  PROBLEM 

As  Halsam  and  Berger  found  in  their  study  (13:1), 
each  year  the  Air  Force  spends  several  hundred  million 
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dollars  updating  its  aircraft  fleet.  The  study  also 
showed  that  an  intricate  management  structure  has  been 
developed  to  review,  approve,  coordinate,  and  implement 
the  total  modification  program. 

One  of  the  major  concerns  regarding  the  Air  Force 
modification  program  has  been  the  increasing  proportion 
of  the  total  budget  which  is  used  for  modification  of  air- 
craft. For  example,  a study  by  Bryant  (4  s 30-35)  showed  j 

that  for  each  of  the  years  from  i960  through  1970  the 

\ 

budget  for  modification  of  aircraft  was  more  than  25  per 
cent  of  the  budget  for  procurement  of  new  aircraft. 

There  is  still  a high  investment  in  aircraft 
modifications.  During  the  period  1972-1976,  the  Air  Force 
invested  over  $500  million  in  modifications  which  were  de- 
signed to  improve  the  reliability  and  maintainability  of 
aircraft.  Although  significant  investments  have  been  made 
in  modifications  to  improve  weapon  system  performance  and 
supportability , no  formal  system  exists  to  evaluate  whether 
actual  benefits  do  materialize  (33). 

The  Air  Force  needs  to  know  if  improvements  in 
logistic  support  are  actually  achieved  from  Class  IV  modi- 
fications. Toward  fulfillment  of  this  need,  an  apparent 
requirement  exists  for  development  of  a technique  for  de- 
termining if  aircraft  reliability,  maintainability,  and 
availability  are  significantly  increased  after  incorpora- 
tion of  Class  IV  modifications  on  aircraft  equipment. 
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JUSTIFICATION  FOR  STUDY 


In  the  April  1976  issue  of  the  Defense  Management 
Journal  (10:1),  it  was  highlighted  that  the  Air  Force  has, 
in  the  last  few  years,  emphasized  that  its  weapon  systems 
are  modified  to  increase  availability  while  keeping  logis- 
tics support  costs  down.  This  concern  for  improving 
availability  and  decreasing  support  costs  is  a result  of 
continuing  reductions  in  the  defense  budget  both  in  abso- 
lute dollar  terms  and  as  a percentage  of  gross  national  pro- 
duct. These  same  fiscal  constraints  limit  the  number  of 
new  weapon  systems  entering  the  active  inventory  as  well  as 
the  number  of  individual  units  of  a system  that  are  pro- 
duced. Paradoxically,  encountering  the  enemy  threat  re- 
quires more  complex  and  more  expensive  weapon  systems. 

Operational  And  Support  Costs 

The  Operational  and  Support  (O&S)  cost  of  Air  Force 
aircraft  has  become  one  of  the  most  urgent  problems  facing 
the  Department  of  Defense  today  (9:24).  During  the  past 
ten  years,  O&S  costs  have  been  increasing  while  appro- 
priated funds  have  shrunk  in  terms  of  real  purchasing  power. 
The  net  result  is  fewer  dollars  with  which  to  develop  and 
procure  technologically  superior  weapon  systems  and  modify 
the  current  aircraft  inventory  (9:24). 

In  the  past,  Air  Force  weapon  systems  have  been 
procured  with  various  problems  of  low  reliability  and  high 
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O&S  costs.  For  example,  the  F-lll  aircraft  was  acquired 
with  specified  equipment  that  was  just  barely  within  the 
state-of-the-art.  Because  of  deficiencies  in  equipment, 
repeated  and  costly  modifications  had  to  be  completed  on 
the  aircraft  after  it  was  introduced  in  the  field  (12:29). 
The  problem  of  low  system  reliability  and  maintainability 
resulted  in  decreased  performance  capability,  increased 
maintenance  manhour  expenditures , increased  failures  and 
aborts,  more  expensive  spare  parts  procurements,  and  costly 
product  improvement  modifications . 

Benefits  Identification 

As  indicated  by  a study  conducted  at  the  Defense 
System  Management  School  in  197^  (^05)*  the  Air  Force  did 
not  at  that  time  have  a systematic  verification  procedure 
for  determining  the  benefits  derived  or  results  achieved 
from  its  aircraft  modification  program.  The  study  stated 
that  for  modification  expenditures  in  excess  of  $5  million, 
a verification  program  was  needed  to  determine  if  the  pro- 
posed improvements  in  equipment  performance  and  reduction 
in  O&S  costs  were  being  achieved. 

A GAO  study  in  October  1977  (32)  also  emphasized 
the  need  for  DOD  to  develop  a system  or  criteria  for  deter- 
mining the  results  obtained  from  its  aircraft  modification 
programs.  The  GAO  study  investigated  whether  the  current 
DOD  emphasis  on  improving  reliability  and  maintainability 
in  its  weapon  systems  has  reduced  operating  and  support 
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costs.  The  study  sought  to  determine  how  cost  savings 
attributable  to  improved  equipment  reliability  and  main- 
tainability could  be  measured.  The  study  concluded  that 
current  data  systems  and  reasonable  criteria  do  not  exist 
for  measuring,  evaluating,  or  tracking  the  effects  of 
reliability/maintainability  improvement  efforts  on  weapon 
system  performance  and  O&S  costs. 

The  purpose  for  developing  a verification  system 
would  be  to  confirm  whether  improvements  do  occur  from  the 
aircraft  Class  IV  modification  programs.  Confirmation  of 
achieved  benefits  would  hopefully  enable  the  Air  Force  to 
obtain  increased  funding  for  its  modification  program, 
whether  it  be  from  existing  Air  Force  sources  or  through 
increased  Congressional  support  for  future  funding  for 
aircraft  modifications  (18). 

As  a part  of  the  verification  system,  the  develop- 
ment of  an  analytical  technique  would  be  extremely  helpful 
to  Air  Force  managers  for  verification  of  the  benefits 
that  are  obtained  from  specific  Class  IV  modifications. 
This  type  of  analysis  would  hopefully  provide  information 
for  justifying  similar  modifications  in  the  future  and 
assist  in  deciding  which  particular  categories  or  types  of 
modifications  provide  the  greatest  improvements  and  bene- 
fits . The  technique  would  also  provide  a check  of  the  in- 
formation included  on  the  AFLC  Form  48.  In  addition,  it 
would  provide  information  which  could  be  used  as  feedback 
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to  the  Configuration  Control  Boards  (CCB)  on  the  effective- 
ness of  approved  modifications.  This  feedback  would  enable 
the  CC3  to  determine  whether  the  AFLC  Forms  ^8  were  accu- 
rately reflecting  the  impact  and  benefits  of  the  future  mod- 
ifications accomplished.  Hence,  the  availability  of  such  a 
technique  would,  hopefully,  allow  for  improvements  in  the 
entire  Air  Force  Class  IV  modification  process. 

CURRENT  EFFORTS 

The  preceding  discussions  may  have  led  the  reader 
to  believe  that  there  are  no  systems  in  being  or  under 
development  to  track  and/or  analyze  the  effect  of  modifica- 
tions designed  to  improve  logistics  support.  This  was  not 
the  intent.  Although  no  tracking  system  has  been  designed 
and  documented  for  Air  Force-wide  application,  there  are 
ongoing  efforts  to  develop  such  a capability.  Some  of  the 
Air  Force's  current  efforts  in  developing  a modification 
tracking  technique  are  described  briefly  in  the  sections 
which  follow. 

KO 51  Modification  Tracking  System 

The  K051  Modification  Tracking  System  is  a sub- 
system of  the  AFLC  Improved  Reliability  of  Operational 
Systems  (IROS)  program.  The  scope  of  the  AFLC  IROS  pro- 
gram includes  the  following: 
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The  composite  effort  of  all  functional  ele- 
ments (that  is,  maintenance,  procurement,  opera- 
tions, engineering,  supply,  materiel  management, 
quality  control,  and  transportation)  to  obtain, 
retain,  and  improve  the  performance  and  reliabil- 
ity of  systems/subsystems/components/parts  on  a 
logistics  effectiveness  basis  . . . [30:10-1], 

Within  the  AFLC  IROS  program  various  data  (such  as 
maintenance  data,  status  data,  and  safety  data)  are  pro- 
cessed through  program  models  to  produce  numerous  displays 
or  output  formats.  One  such  set  of  displays  is  the 
K051-BTXL  products  which  are  formally  entitled  "Audit  of 
Class  IV  Modifications  by  Mod  Number"  and  referred  to  here- 
in as  the  K051  Class  IV  Modification  Tracking  System.  This 
system  has  not  been  fully  documented  and  was  not  briefed 
at  the  October  1977  Product  Improvement  Program  meeting 
because  "it  contains  the  inherent  assumption  that  a modi- 
fication affects  only  one  piece  of  hardware  (WUC)  in  one 
weapon  system"  (29:5).  This  is  because  failure,  mainten- 
ance, and  status  information  are  aggregated  by  work  unit 
code  (WUC)  in  the  maintenance  data  collection  system  from 
which  the  IROS  program  draws  its  data  (30:10-1).  This 
deficiency  would  be  present  in  any  tracking  or  analysis 
system  which  depended  upon  the  maintenance  data  collection 
system  for  its  input  data.  Another  apparent  deficiency  of 
this  system  is  that  it  only  allows  one  to  determine  the 
overall  contribution  of  the  WUC  to  logistics  support  cost 
or  availability  degradation.  It  does  not  permit  separation 
of  the  variables  which  may  be  causing  system  availability 
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degradation.  For  example,  an  item  (WUC)  may  be  modified 
because  of  a high  contribution  to  system  Not  Mission  Cap- 
able Maintenance  (NMCM)  time.  If,  after  modification, 
there  was  an  increased  contribution  by  this  WUC  to  system 
availability  degradation  because  of  nonavailability  of 
spares,  any  improvements  made  would  be  masked. 

Air  Force  Academy  Effort 

Mitchell  and  Carter  conducted  a research  effort  and 
reported  their  results  in  February  1978  in  which  the  objec- 
tive was  to  investigate  the  modification  history  on  the  F-4 
aircraft  and  to  . . model  reliability  improvement  at  the 
aircraft  level  [20:iii]” . The  draft  report  of  this  effort 
indicated  that  insufficient  data  were  available  to  meet  the 
research  objective.  However,  the  data  were  sufficient  for 
doing  some  developmental  work  on  a modification  tracking 
and  analysis  system. 

Essentially,  the  procedure  developed  by  Mitchell 
and  Carter  used  a "matched  pairs”  statistical  test  to  test 
for  differences  in  the  means  of  selected  variables  (mainte- 
nance actions,  maintenance  manhours,  and  failures)  applied 
to  a specified  WUC  before  and  after  modification  (20*.9-l4). 
This  approach  was  appealing  because  it  allowed  individual 
assessment  of  variables  which  were  indicative  of  system  re- 
liability and  maintainability.  Some  of  Mitchell  and  Carter’s 
procedures  for  adapting  the  0098  data  system  to  track  air- 
craft performance  variables  have  been  used  in  this  study. 
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PRAM  Program 

In  July  1 975 » the  Air  Force  established  the  Produc- 
tivity, Reliability,  Availability,  and  Maintainability  (PRAM) 
program.  The  primary  objective  of  this  program  was  to  focus 
management  attention  and  funds  in  a concerted  effort  to  re- 
duce operational  weapon  systems  support  costs  by  innovative 
investment  in  cost-reducing  opportunities.  One  anticipated 
benefit  would  be  improved  weapon  system  reliability,  main- 
tainability, and  supportability  (9»24). 

Currently,  the  PRAM  office  maintains  a data  source 
of  information  called  LIST  (Logistics  Investment  Screening 
Technique)  (11).  The  basic  strategy  and  intent  of  the  data 
source  are  to  provide  a listing  by  work  unit  code  of  air- 
craft equipment  items  that  possess  a high  benefit-to- 
investment  ratio  for  reliability  improvement.  The  purpose 
for  establishing  the  data  was  to  allow  equipment  and  weapon 
system  managers  to  isolate  equipment  items  with  the  greatest 
opportunities  for  benefits  for  modification  dollar  invested. 
Furthermore,  it  should  help  identify  those  items  that  may 
improve  weapon  system  availability,  reduce  aborts,  and  re- 
duce logistics  support  costs  (11:8).  Because  this  technique 
has  just  recently  been  developed,  its  impact  on  the  modifi- 
cation process  can  not  be  evaluated. 
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CURRENT  PROBLEMS 


In  an  October  1977  article  in  the  Defense  Management 
Journal , Kern  of  Hughes  Aircraft  discussed  the  problems  that 
the  Air  Force  experiences  when  comparing  equipment  reliabil- 
ity which  is  based  on  actual  field  operation  with  the  equip- 
ment reliability  obtained  from  laboratory  tests . The 
article  emphasized  that  the  disparity  between  the  values 
can  be  attributed  to  differences  in  definitions  of  terms, 
operational  factors,  and  environmental  factors  (l4-:l8). 

The  problems  that  Kern  discussed  are  generated  by 
the  inconsistencies  in  the  criteria  used  for  assessing 
field  data  generated  by  various  aircraft  systems.  These 
inconsistencies  result,  primarily,  from  differences  in  def- 
initions used  for  such  things  as  time  and  failure  (l4sl8). 
Air  Force  Logistics  Command  (AFLC)  relies  primarily  on 
weapon  system  flying  hours,  rather  than  equipment  operating 
hours,  as  the  time  base  in  calculating  mean  time  between 
failures . 

The  Maintenance  Requirements  Data  System  (G098) 
which  will  be  used  in  this  study  as  the  source  of  historical 
data  on  aircraft  performance  does  not  maintain  equipment 
operating  time  and  the  information  is  difficult  to  obtain 
since  total  operating  time  is  only  maintained  on  a few 
selected  equipment  items.  For  these  reasons,  equipment 
operating  time  accrued  during  ground  operation  or  during 
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operational  checkouts  and  maintenance  is  not  analyzed  in 
this  study.  It  will  be  assumed  that  flying  hours  and  sorties 
will  provide  relative  measures  of  total  equipment  operating 
time . 

The  failure  data  in  G098  also  includes  some  infor- 
mation that  would  not  be  considered  as  relevant  for  relia- 
bility test  assessment  purposes  (29).  For  example,  as  Kern 
explained  in  his  article,  an  item  may  be  removed  for  ex- 
ternal causes  such  as  improper  troubleshooting  or  faulty 
test  equipment.  This  will  result  in  mistakenly  replacing 
the  equipment  which  really  has  not  failed.  In  addition, 
the  equipment  item  may  be  removed  and  replaced  for  other 
externally  caused  factors  such  as  broken,  bent,  loose, 
cracked,  or  missing  components  that  do  not  affect  the 
operational  status  of  the  item.  These  replacement  actions 
would  not  normally  be  included  in  the  assessment  of  the  in- 
herent equipment  reliability  but  these  actions  will  be  in- 
cluded in  the  failure  data  maintained  in  the  G098  system. 
These  types  of  failures  will  also  result  in  maintenance 
actions  and  maintenance  manhours  being  documented  against 
a system. 

SCOPE 

This  research  is  limited  to  aircraft  modifications 
which  fail  in  the  correction  of  material  deficiencies 
(Class  IV)  category.  As  described  previously,  the 
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objectives  of  Class  IV  modifications  can  be  summarized  ass 
(1)  improved  maintainability,  (2)  improved  service  life, 

(3)  improved  logistics  support — implying  improved  relia- 
bility, and  (4)  reduced  costs. 

Class  I and  II  modifications  were  not  considered  in 
this  research  because  they  pertain  to  a small  number  of 
aircraft  for  a short  period  of  time  and  generally  do  not 
address  reliability.  Class  III  modifications  were  not  con- 
sidered because  they  pertain  only  to  aircraft  in  production. 
Class  V modifications  were  not  considered  because  they  pro- 
vide new  capabilities . 

Within  the  Class  IV  category,  the  data  of  concern 
in  this  study  were  limited  to  those  that  provide  indica- 
tions of  changes  in  equipment  reliability,  maintainability, 
and  availability. 

RESEARCH  QUESTION 

Can  a systematic  technique  which  uses  available 
systems  data  be  developed  to  determine  whether  or  not  the 
stated  purposes  appearing  on  the  AFLC  Forms  48  as  part  of 
the  original  justification  for  Class  IV  modifications  have 
been  achieved? 

RESEARCH  OBJECTIVES 

The  primary  objective  of  this  research  was  to  de- 
velop a technique  for  assessing  the  effectiveness  of 
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logistics  support  (Class  IV)  modifications.  A secondary 
objective  was  to  use  the  developed  technique  to  assess  the 
effectiveness  of  selected  Class  IV  modifications. 


SUMMARY 

This  chapter  has  provided  the  background  and  justi- 
fication for  this  study.  It  has  briefly  explained  the  prob- 
lem and  some  current  Air  Force  efforts  in  resolving  it. 

Some  aspects  of  the  current  aircraft  modification  process 

' 

were  explained.  Finally,  the  scope,  objectives,  and  re- 
search question  were  provided. 

Chapter  2 will  discuss  the  data  collection  process 
and  the  variables  utilized  in  this  research.  Chapter  3 
will  describe  the  technique  and  the  steps  that  were  used 
in  analyzing  modification  data.  Chapter  4 will  present  the 
application  of  the  technique  to  selected  modifications  and 
Chapter  5 will  include  the  summary,  conclusions,  and 
recommendations  for  future  research. 
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Chapter  2 


OVERVIEW  OF  DATA  AND  VARIABLES 

This  chapter  outlines  the  procedures  employed  by 
the  authors  to  collect  the  data  used  in  this  research 
effort,  describes  the  sources  of  data,  and  provides  the 
criteria  used  to  select  the  modifications  used  in  this 
study.  Additionally,  it  defines  the  variables  of  interest 
in  this  study,  and  identifies  some  variables  which  may  af- 
fect logistics  support  but  which  were  excluded  from  the 
analysis . 

DATA  COLLECTION 

Initial  development  of  the  technique  described  in 
this  study  was  done  on  data  used  by  Mitchell  and  Carter  in 
their  study  (20).  These  data  did  not  provide  any  indica- 
tion of  the  effects  of  a modification  on  availability.  It 
was  therefore  determined  that  other  modifications  used 
should  include  Not  Operationally  Ready  Maintenance  (NORM) 
and  Not  Operationally  Ready  Supply  (NORS)  time  as  measures 
of  availability. 

The  authors  had  initially  assumed  that  complete 
records  of  past  modifications  were  maintained  at  Head- 
quarters AFLC . Interviews  with  personnel  of  the  AFLC 
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Materiel  Support  Division  (AFLC/LOAT)  disclosed  that  Head- 
quarters AFLC  only  maintained  records  for  active  modifica- 
tions approved  at  that  level.  It  was  therefore  concluded 
that  an  Air  Logistics  Center  (A1C)  would  be  a more  appro- 
priate source  for  identification  of  modifications  to  be 
studied . 

In  light  of  the  above,  a visit  was  made  to  the 
Policy  and  Programs  Section  (MMMMP)  at  Wamer-Robins  ALC. 

From  a review  of  a card  file  maintained  by  this  office, 

58  modifications  were  selected  as  potential  candidates  for 
further  study.  The  criteria  used  in  this  initial  selection 
were  as  follows: 

1 . The  modification  had  been  completed  for  a mini- 
mum of  six  months.  This  criteria  insured  that  18  months  of 
operational  time  had  been  gained  on  a representative  number 
of  modified  aircraft.  Eighteen  months  of  operational  data 
were  required  to  minimize  the  effects  of  factors  not  analyzed 
in  this  research. 

2.  The  modification  was  a Class  IVA,  IVB  or  IVC. 
Initially,  this  criteria  required  Class  IVC  modifications 
only.  However,  due  to  the  paucity  of  data  on  the  Class  IVC 
modifications,  the  criteria  were  relaxed  to  allow  for  the 
inclusion  of  Class  IVA  or  Class  IVB  modifications  which  had 
improvement  of  one  of  the  logistics  support  elements  as  a 
strong  secondary  objective. 
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The  Configuration  Control  Board  Item  Records  ( AFLC 
Forms  48)  for  the  58  modifications  initially  identified 
were  then  reviewed.  The  criteria  used  to  select  modifica- 
tions for  further  review  and/or  analysis  were: 

1.  A stated  purpose  of  the  modification  was  to  re- 
duce manhours,  maintenance  actions,  NORM  or  NORS  hours, 
failures,  or  to  improve  reliability. 

2.  The  AFLC  Form  48  contained  a data  code  (for  air- 
craft controlled  by  the  Advanced  Configuration  Control  Sys- 
tem) . This  was  required  to  extract  the  date  individual 
aircraft  were  modified  from  the  historical  records. 

3.  The  AFLC  Form  48  contained  a work  unit  code 
which  was  required  to  extract  performance  data  from  the 
G098  data  system  (explained  in  greater  detail  later  in  this 
paper) . 

Only  15  modifications  survived  this  second  screen- 
ing. Five  of  these  were  modifications  of  equipment/systems 
on  the  HH-53  helicopter  for  which  no  data  were  available  in 
the  G098.  The  predominant  reason  for  elimination  of  the 
initial  43  modifications  was  the  absence  of  a work  unit 
code  on  the  AFLC  Form  48. 

The  modification  selection  method  required  the 
personal  involvement  of  one  of  the  authors  and  time  con- 
straints prevented  multiple  A1C  visits.  Therefore  only 
C-130,  C-141A,  F-4C/D/E,  and  F-106  aircraft  are  represented 
in  this  study. 
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Historical  tapes,  containing  the  serial  numbers  of 
aircraft  receiving  the  modifications  and  the  dates  the 
modifications  were  accomplished,  were  provided  to  the 
Scientific  and  Engineering  Systems  Unit  (ACDCK)  at  San 
Antonio  ALC  by  personnel  from  the  Requirements  Section 
(ACDCH)  at  Wamer-Robins  ALC.  These  data  were  then  merged 
with  data  contained  in  the  Maintenance  Requirements  Data 
System  by  personnel  from  San  Antonio  ALC  (ACDCK)  who  used 
a program  which  they  maintain.  The  data  were  then  provided 
to  the  authors  on  punch  cards . 

DATA  COLLECTION  STEPS 

The  steps  in  the  data  collection  process  are  shown 
in  Figure  2-1 . Numbers  in  the  figure  are  coded  to  the 
steps  described  below. 

Step  1,  Determine  Data  Requirements 

The  authors  were  interested  in  determining  which, 
if  any,  of  the  previously  described  logistics  support 
elements  were  improved  by  the  selected  modifications  for 
the  purpose  of  developing  an  evaluation  technique.  There- 
fore, all  data  considered  to  be  relevant  to  this  determina- 
tion were  required.  The  order  of  this  step  and  the  next 
could  logically  be  reversed  if  one  were  only  attempting 
to  evaluate  the  effectiveness  of  a specific  modification. 
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Fig.  2-1.  Data-Gathering  Process 
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Step  2.  Select  Modifications  For  Study 

The  AFLC  Form  48  was  used  as  the  vehicle  for  the 
selection  process.  This  form  provided  a simple  data  source 
which  would  yield  data  on  the  purpose  and  identification  of 
the  modification  as  well  as  the  work  unit  code  (WUC)  of  the 
item  modified.  These  data  items  were  necessary  for  later 
data  retrievals  and  analyses.  Any  other  data  source  from 
which  the  above  data  could  be  derived  would  serve  this 
purpose . 

Step  3.  Determine  Monthly  Data  Requirements 

Eighteen  months  of  data  were  selected  for  this 
study.  It  was  assumed  that  eighteen  months  of  activity 
would  dampen  the  effects  of  factors  not  considered  on  the 
logistics  support  factors  being  evaluated. 

Step  4.  Determine  Modification  Identification  Data 

It  was  necessary  to  have  the  WUC,  serial  numbers  of 
aircraft  modified,  and  date  each  aircraft  was  modified  in 
order  to  extract  the  data  related  to  the  logistics  support 
elements  from  the  G098  data  system.  The  WUC  shown  on  the 
AFLC  Form  48  was  the  only  data  with  which  the  work  performed 
could  be  related  to  the  item  modified. 

Step  5.  Provide  Identification  Data  To  San  Antonio  ALC 
This  step  was  accomplished  for  the  authors  by 
personnel  from  Wamer-Robins  ALC  (ACDCH)  who  provided 
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San  Antonio  ALC  ( ACDCH)  with  copies  of  the  D047  (C-130 
modifications)  and  D057G  (C-l4l  modifications)  historical 
tapes.  The  data  code  for  the  modification  allowed  the  San 
Antonio  ALC  personnel  to  select  the  aircraft  serial  numbers 
and  dates  of  modification  from  the  historical  tapes.  The 
WUC  and  number  of  months  of  data  required,  allowed  ex- 
traction of  failure,  maintenance  actions,  manhour  consump- 
tion, and  contribution  to  nonavailability  data  (as  related 
to  the  identified  WUC)  from  the  G098  data  system. 

Step  6.  Extract  Data  From  The  G098  Data  System 

This  step  was  accomplished  by  San  Antonio  ALC 
(ACDCK)  personnel  with  a program  developed  and  maintained 
by  them.  Essentially,  the  program  allows  compilation  of 
logistics  performance  data  for  a specified  period  of  time 
before  and  after  a modification. 

Step  7.  Receive,  Input,  And  Review  Data 

The  data  were  received  from  San  Antonio  ALC  (ACDCK) 
in  punched  card  form  as  a matter  of  convenience  to  the 
authors.  Tape  output  could  have  just  as  easily  been  pro- 
vided. The  data  were  then  reviewed  to  ensure  that  the 
format  was  amenable  to  analyses  and  served  as  input  to 
programs  developed  locally. 
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Step  8.  Formatted  Data  For  Analysis 

This  was  the  output  step  of  the  data  gathering 
process.  The  data  were  now  ready  to  be  used  as  the  input 
to  the  analysis  process. 

AVAILABLE  DATA  SOURCES 

A review  of  current  Air  Force  literature  (30,  31) 
revealed  that  there  are  numerous  sources  of  data  on  histor- 
ical performance  of  aircraft  in  the  Air  Force  inventory. 
However,  the  most  extensive  source  of  data  on  system/equip- 
ment  operational  performance  is  provided  in  the  Air  Force 
by  two  primary  data  reporting  systems . One  of  these  is  the 
maintenance  data  collection  system  specified  in  the 
00-20  Series  Technical  Orders  and  Air  Force  Manual  ( AFM) 
66-1.  The  other  is  the  standard  aerospace  and  equipment 
status  reports  specified  in  AFM  65-110. 

The  Maintenance  Data  Collection  System  (MDCS)  is 
primarily  directed  toward  the  recording  and  reporting  of 
base  maintenance  activity  but  it  is  also  required  for  con- 
tractor and  depot  level  maintenance.  Basically,  the  MDCS 
provides  the  following  datat 

1 . Work  done . 

2.  Why  the  maintenance  was  necessary. 

3.  When  the  discrepancy  was  discovered. 

4.  Who  did  the  work. 

5.  How  many  manhours  did  it  take. 

The  above  information  is  recorded  by  the  technician 
when  he  accomplishes  a job.  The  technician  specifies  the 
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equipment  item  that  the  work  was  accomplished  on  by  use  of 
a 5-digit  Work  Unit  Code  (WUC).  AFM  65-110  data  provide 
information  on  equipment  inventory,  operating  time,  and 
availability  (operational  status)  and  are  also  provided  from 
base  level  managers. 

These  two  data  systems  have  been  merged  into  the 
G098  data  system  which  is  maintained  by  Kelly  AFB.  In  this 
study_the  G098  system  was  selected  because  it  was  determined 
that  it  is  the  most  complete  data  storage  and  retrieval  sys- 
tem available  in  the  Air  Force.  The  G098  system  is  main- 
tained, however,  only  on  selected  aircraft.  It  is  important, 
then,  to  insure  that  in  the  data  collection  process  that 
aircraft  selected  for  analysis  were  currently  maintained 
in  the  G098  system. 

PROBLEMS  WITH  DATA 


The  AFM  66-1  information  system  has  been  criticized 
for  its  inaccuracy.  In  a recent  study  by  Badalamente  and 
Clark,  it  was  stated: 

Perhaps  the  most  often  cited  shortcoming  of  the 
MDC  system  is  the  inaccuracies  in  its  data  and  the 
resultant  unreliability  of  its  output.  The  criticism 
' was  first  formalized  in  a 1965  IG  Report,  which 

stated  that  there  was  25  to  400  percent  padding  in 
manhour  data  and  a 40  percent  error  rate  in  MDC  pro- 
duct improvement  data.  . . . Numerous  attempts  have 
been  made  over  the  years  to  improve  MDC  accuracy  and 
there  has  been  continuous  and  sometimes  brutal  high- 
level  emphasis  on  accurate  documentation  at  base 
level.  Nevertheless,  the  perceived  if  not  actual, 
accuracy  and  reliability  of  the  MDC  system  remains 
low  [1:12], 
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In  the  conclusion  of  their  study  the  authors  pro- 
vided some  suggestions  on  how  to  improve  the  MDC  system. 
Basically,  they  concluded  that  the  MDC  system  in  its  current 
form  has  the  potential  to  support  decision  making  at  various 
levels  of  the  logistics  management  function  (1:22-23). 

Other  studies  have  been  completed  that  also  com- 
mented on  the  accuracy  of  the  66-1  MDC  system.  A RAND  study 
in  1975  (8)  that  evaluated  the  Air  Force  IROS  system  as  an 
accurate  estimator  of  logistics  support  costs  commented  on 
the  accuracy  of  MDC  data  used  in  IROS: 

In  almost  any  cost  analysis,  comparative  or 
otherwise,  there  are  limitations  imposed  by  the  lack 
of  adequate  data,  and  this  study  is  no  exception. 

The  basic  data  systems  that  we  relied  on  for  our 
reference  costs  are  not  perfect.  These  data  pro- 
ducts contain  ambiguities  and  in  some  instances  are 
possibly  incomplete.  However,  they  do  reflect  di- 
rect recordings  of  the  maintenance  and  material 
resources  consumed  to  support  the  weapon  system  of 
interest  [8:4]. 

The  RAND  study  commented  on  some  of  the  specific 
deficiencies  in  the  MDC  system.  The  study  indicated 
that  the  MDC  data  system  does  not  consistently  record  all 
relevant  maintenance  actions.  The  study  referenced  another 
RAND  report  (25)  that  found  in  1973  that  base -level  re- 
pair actions  were  frequently  lost  or  not  indicated  in  the 
MDC  system.  The  report  stated: 

For  example,  within  a job  control  number  a com- 
mon sequence  of  actions  is  a pull,  a repair,  and  a 
replace.  When  one  component  of  this  sequence  is 
missing  in  the  MDC  66-1  report  (e.g.,  if  only  a re- 
move and  replace  were  recorded),  there  is  a logical 
error  of  omission.  These  data  deficiencies  in  MDC 
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66-1  are  seemingly  passed  on  to  the  K051  [IROS]  re- 
ports, since  there  is  no  apparent  input  data  logical 
sequence  quality  check.  On  the  other  hand,  the  MDC 
system  is  a self-recording  system  and  one  has  also 
to  check  for  exaggerated  numbers  of  maintenance 
hours  reported — numbers  incompatible  with  the  re- 
sources available  for  maintenance  [8tl7]. 

A 1978  study  authored  by  Stanhagen  (24)  described 
an  investigation  into  the  accuracy  of  MDC  data  used  in  the 
Air  Force.  The  report  emphasized  that  in  addressing  the 
accuracy  of  the  data  there  are  two  basic  methods  for  errors 
to  develop.  The  first  being  an  error  in  the  technical 
competence  of  the  mechanic  who  supposedly  corrects  the  dis- 
crepancy. The  mechanic  may  not  complete  the  proper  trouble- 
shooting and  therefore  repairs  the  wrong  item.  This  would 
result  in  a documentation  error  where  the  wrong  WUC  is 
credited  with  causing  the  discrepancies  and  having  erron- 
eous maintenance  data  coded  against  it.  This  type  of  error 
can  occur  in  any  type  of  collection  system  and  can  be  re- 
duced by  properly  training  and  supervising  the  mechanics. 

The  other  type  of  error  results  from  the  mechanic  who  re- 
pairs the  correct  item  but  copies  down  the  wrong  WTJC, 
action  taken  code,  how  malfunctioned  code,  and  so  forth. 

In  this  study,  168  Air  Force  aircraft  mechanics 
were  identified  as  participants  in  a test  program.  The 
mechanics  were  given  specific  tasks  to  complete  on  C-135 
aircraft.  The  work  they  accomplished  was  then  evaluated 
by  a panel  of  six  "experts"  to  determine  how  accurately  the 
mechanics  coded  their  work.  The  final  conclusion  from  this 
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study  was  that  MDC  "data  is  indeed  accurate  and  capable  of  j 

providing  an  effective  basis  for  logistics  decision  making  1 

[24:57]."  | 

t 

A briefing  at  the  October  1977  Product  Improvement  j 

Program  (PIP)  Problem  Definition  meeting  at  Wright-Patterson  • 

AFB  also  addressed  the  validity  of  data  collected  by  ' 

AFM  66-1.  The  position  of  the  representatives  from  Warner-  ! 

Robins  ALC  was  that  they  realized  that  the  data  may  be  in- 
accurate to  some  extent.  However,  they  indicated  that  "there 

\ 

appears  to  be  no  alternate  source  of  data  which  can  provide 
comparable  information  [29:87]."  The  briefing  continued: 

When  component  or  system  problems  arise,  66-1 
data  including  the  "on  equipment"  information, 
provides  valuable  insight  into  the  problems  resolu- 
tion and  often  provides  a substantial  justification 
for  any  follow-on  modification  proposal.  WR-ALC 
position  [is  to]  retain  the  collection  of  on  equip- 
ment data  [29:87J. 

DATA  SOURCES  UTILIZED 

The  data  used  to  measure  the  variables  in  this  pro- 
ject were  initially  extracted  from  the  Advanced  Configura-  I 

tion  Management  System  (D057G),  The  Standard  Configuration  1 

Management  System  (D047),  and  the  Maintenance  Requirements 
Data  System  (G098)  by  personnel  assigned  to  the  data  auto- 
mation center  at  Kelly  AFB.  The  D057G  and  the  D047  contain 
modification  data  and  allows  one  to  determine  the  date  a 
specific  aircraft  was  modified  (20:6).  The  G098  data  sys- 
tem was  designed  in  1970  to  provide  a data  base  containing 
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historical  data  which  could  serve  as  the  basis  for  the 
engineering  determination  of  content  of  and  interval  be- 
tween programmed  depot  maintenance.  The  file  contains 
aircraft  maintenance,  inventory,  utilization,  and  status 
information  for  each  aircraft  by  tail  number  since  1966 
(29).  The  data  from  the  D057G  and  G098  were  then  merged 
at  Kelly  AFB  to  provide  the  information  to  analyze  the 
effectiveness  of  a specific  modification. 

A list  was  compiled  of  aircraft  by  tail  number  that 
had  been  modified.  Data  were  provided  from  the  G098  data 
system  for  a specified  period  of  time  before  and  after  the 
modification  on  the  following  variables  for  each  tail 
numbers  (1)  flying  hours,  (2)  sorties,  (3)  maintenance 
actions,  (4)  maintenance  manhours,  (5)  failures,  (6)  Not 
Operationally  Ready  Maintenance  (NORM)*,  and  (7)  Not  Opera- 
tionally Ready  Supply  (NORS)*. 

The  last  variables  (maintenance  actions,  manhours, 
failures,  NORM,  and  NORS)  contain  data  on  the  modified  part 
identified  by  a specific  Work  Unit  Code  (WUC).  It  should 
be  emphasized  that  only  maintenance  data  related  to  this 
specific  WUC  are  of  interest  for  the  analyses  accomplished 
in  this  study.  The  data  compiled  in  G098  are  obtained  from 


•These  two  variables  were  changed  to  Not  Mission 
Capable  Maintenance  and  Not  Mission  Capable  Supply  on  1 
October  1977*  The  old  nomenclature  is  used  in  the  analyses 
to  maintain  consistency  with  the  terminology  in  use  at  the 
time  of  modification  development  and  approval. 
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work  reports  documented  in  the  field  by  maintenance  techni- 
cians. Each  job  accomplished  in  the  field  must  identify 
the  WUC  or  equipment  item  that  the  job  related  to.  This 
identification  system  allows  specific  tracking  of  how  much 
logistic  support  was  required  to  maintain  a specific  equip- 
ment item. 


POPULATION 

The  population  for  which  the  technique  was  developed 
includes  all  modifications  designed  to  improve  logistics 
support.  .» 

SAMPLE 

The  sample  against  which  the  technique  was  applied 
consisted  of  three  Class  IVC,  one  Class  IVA, and  four  Class 
IVB  modifications  selected  by  the  authors  in  accordance  with 
the  data  gathering  plan  presented  earlier.  One  F-l 06  Class 
IVB  modification  was  selected  by  San  Antonio  ALC  ( ACDCK) 
personnel  to  test  the  data  retrieval  program  and  was  sub- 
sequently used  in  the  analysis  process.  In  addition,  the 
data  on  five  modifications  (four  Class  IVA  and  one  Class 
IVB)  used  by  Mitchell  and  Carter  in  their  research  (20) 
were  also  analyzed. 


33 


VARIABLES 


In  discussing  product  improvements  in  the  Air  Force 
logistic  managers  should  become  concerned  about  analyzing 
such  parameters  as  reliability  and  maintainability  which  are 
determinants  of  logistic  support  requirements.  These  param- 
eters have  considerable  importance  when  designing  a new 
weapon  system.  In  the  life  cycle  cost  approach  to  Air  Force 
weapon  system  acquisition,  considerable  emphasis  should  be 
placed  on  these  parameters  in  an  attempt  to  reduce  opera- 
tional and  support  costs. 

In  1976,  the  Department  of  Defense  began  to  estab- 
lish reliability  goals  and  thresholds  for  new  weapon  systems 
in  the  Decision  Coordinating  Paper  (DCP).  Actual  equipment 
reliability  values  will  be  calculated  using  field  data 
collection  ground  rules  and  procedures  (19:6).  Therefore, 
it  has  become  important  that  logistic  support  parameters  be 
understood  in  terms  of  variables  maintained  on  Air  Force 
weapon  systems  in  current  maintenance  data  information 
systems . 

Variables  Used 

The  variables,  their  definitions,  and  their  data 
level  analyzed  in  this  study  are  as  follows: 

Failure . A ratio  level,  discrete,  infinite  variable 
which  is  reported  for  those  maintenance  actions  on  the 
equipment  under  consideration  charged  with  a failure 
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malfunction,  a repair  or  replacement  action,  and  which  were 
discovered  by  the  aircrew. 

Maintenance  manhours . A ratio  level,  discrete,  in- 
finite variable  which  is  reported  by  the  mechanic  at  the 
time  of  his  maintenance  action.  These  data  indicate  the 
number  of  manhours  extended  by  the  mechanic  to  correct  a 
specific  reported  failure. 

Maintenance  action.  A ratio  level,  discrete,  in- 
finite variable  which  is  considered  as  any  action  taken  on 
the  equipment  under  consideration  for  which  the  mechanic 
charges  some  expenditure  of  time. 

Flying  hours . A ratio  level,  discrete,  infinite 
variable  which  is  the  time,  in  hours,  between  aircraft 
takeoff  and  an  associated  landing.  For  any  period  of  time, 
the  flying  hours  are  given  as  the  sum  of  the  flying  hours 
from  all  sorties  for  that  period. 

Sortie . A ratio  level,  discrete,  infinite  variable 
which  is  reported  as  am  event  consisting  of  one  takeoff  and 
associated  landing. 

NORM.  A ratio  level,  discrete,  infinite  variable 
which  is  the  time,  in  hours,  that  a weapon  system  is  un- 
available for  its  assigned  mission  due  to  a particular 
maintenance  action  being  accomplished  on  equipment. 

NORS . A ratio  level,  discrete,  infinite  variable 
which  is  the  time,  in  hours,  that  a weapon  system  is 
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unavailable  for  its  assigned  mission  due  to  lack  of  a 
required  part. 

Using  the  above  variables  currently  maintained  in 
maintenance  data  systems  the  following  relationships  were 
assumed  and  used  in  this  study  to  help  describe  some  of  the 
parameters  that  affect  weapon  system  performance  and  logis- 
tic supports 

1.  Failures  and  maintenance  actions  provided  an 
indication  of  equipment  reliability. 

2.  Maintenance  manhours  provide  sin  indication  of 
equipment  maintainability. 

3.  Flying  hours  and  sorties  provide  an  indication 
of  weapon  system  activity. 

4.  NORM  and  NORS  hours  provide  an  indication  of 
system  availability  degradation  attributable  to  the  mod- 
ified equipment. 

Additional  Factors 

In  the  current  literature,  there  is  considerable 
concern  about  what  effects  different  factors  have  on  weapon 
system  performance  (7t  14).  Kern  described  some  of  these  in 
his  article  under  the  general  headings  of  operational  and 
environmental  factors . The  following  factors  have  been 
discussed  in  the  current  literature  (7,14,19,31)  as  possible 
factors  affecting  not  only  logistic  support  but  system 
performance  in  general: 
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1.  Material  Handling:  The  amount  a given  item  is 
subjected  to  transportation  and  maintenance  handling  in  terms 
of  removal  rate  from  the  aircraft  for  repair  or  bench  checks. 

2.  Base  Maintenance  Policy:  Local  base  policies 
concerning  cannibalization,  operational  checks,  shop  repair 
policy,  training  and  skill  levels  of  mechanics,  and  other 
base  unique  maintenance  policies. 

3.  Environmental  Influences:  Climatic  and  tempera- 
ture variations,  humidity,  dust,  and  other  conditions  unique  , 

to  a particular  base . 

4.  Mission:  Unique  mission  of  the  base  may  require 

t 

greater  utilization  of  a particular  system  or  component. 

Unique  missions  cause  different  stresses  to  be  placed  on  j 

particular  systems. 

5.  Facilities:  Hangar  facilities,  shop  test  equip- 
ment, condition  of  ramps  and  taxiways,  outside  lighting,  and  1 

other  facilities  determine  type  and  quality  of  maintenance  ! 

that  can  be  accomplished  on  a system. 

In  this  study,  it  became  apparent  that  these  factors 
were  not  quantifiable  from  the  available  data  and  were  ex- 
cluded from  the  analysis.  For  example,  a system  could  be 
inoperative  and  go  undetected  because  no  demand  was  placed 
on  it  for  the  particular  mission  being  flown.  This  ob- 
viously would  affect  the  number  of  failures,  maintenance 
manhours,  and  maintenance  actions  documented  against  the 
sys tem. 
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SUMMARY 


This  chapter  has  described  the  data  collection  * 

I 

process  that  was  used  in  this  study.  The  criteria  used  in 
selecting  the  modifications  analyzed  in  this  research  were 

i 

also  explained.  Available  data  sources  and  their  inherent 

problem  areas  were  discussed.  Finally,  the  variables  used  j 

in  the  analyses  were  defined  along  with  the  factors  not 
considered  in  this  study.  The  discussions  in  Chapter  3 

will  address  the  technique  development.  i 
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Chapter  3 


TECHNIQUE  DEVELOPMENT 

The  purpose  of  this  chapter  is  to  provide  a 
detailed  description  of  the  analytical  technique  employed 
in  this  research  effort,  the  statistical  tests  utilized, 
and  the  rationale  for  the  use  of  the  technique. 

APPROACH 

The  general  premise  of  this  research  effort  is 
that  the  mean  values  (before  and  after  a modification)  of 
the  variables  under  consideration  can  be  compared  to  pro- 
vide an  indication  of  the  effects  of  the  modification.  If 
there  is  no  significant  difference  between  aircraft  activ- 
ity before  and  after  the  modification,  the  comparisons  can 
be  made  directly.  However,  if  there  is  a significant  dif- 
ference in  aircraft  activity  during  the  two  periods,  the 
variable  value  must  be  normalized  (divided  by  an  appropriate 
activity  measure)  before  the  comparison  is  made. 

Aircraft  and  engine  performance  is  commonly  meas- 
ured in  flying  hours . These  same  flying  hours  are  also 
generally  applied  to  the  aircraft  equipment  as  a measure 
of  operating  time.  Many  studies  have  been  made  of  flying 
hour/sortie  relationships.  These  studies  indicate  that 
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in  some  cases  sorties  provide  a more  definitive  perform- 
ance measurement  (31:18). 

For  these  reasons,  the  data  describing  the  vari- 
ables in  this  research  effort  were  compared  directly  and 
were  also  compared  after  being  normalized  by  flying  hours 
and  sorties . The  steps  in  the  process  are  displayed  in 
Figure  3-1 • The  normalization  process  was  included  in  the 
computer  programs  used  to  perform  the  statistical  analyses 
(described  later  in  this  chapter). 

The  following  steps  as  shown  in  Figure  3-1  were 
used  in  the  analysis  of  data. 

Step  1 . Establish  Criteria 

The  criteria  that  were  used  in  this  study  to  de- 
termine the  effectiveness  of  the  modification  being  analyzed 
were  developed  in  this  step.  A modification  was  determined 
to  be  effective  if  it  achieved  the  purpose  as  indicated  in 
the  modification's  approval  justification  on  the  AFLC  Form 
48  and  no  other  variables  increased  after  the  modification. 

Step  2.  Perform  Statistical  Analyses  Of  Data 

The  following  paragraphs  describe  the  statistical 
analyses  that  were  utilized  in  this  technique.  Utilization 
of  these  particular  statistical  techniques  are  described  in 
greater  detail  later  in  this  study.  The  selection  of  the 
statistical  hypotheses,  test  statistics,  and  decision  rules 
will  be  determined  by  the  researcher  and  will  depend  on  the 
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particular  variables  being  analyzed  and  the  level  of  sig- 
nificance desired. 

Step  2a.  Determination  Of  Data  Distribution 

The  first  statistical  test  was  employed  to  determine 
if  the  data  being  analyzed  were  normally  distributed.  The 
program  utilized  the  Kolraogorov-Smirnov  (K-S)  Goodness-of- 
Fit  Test  to  conclude  whether  the  data  were  normally  dis- 
tributed . 

Step  2b.  Selection  Of  Appropriate  Difference  Test 

From  the  conclusions  of  the  K-S  tests  it  was  deter- 
mined whether  the  parametric  Paired  t-Test  or  the  non- 
parametric  Wilcoxon  Matched -Pairs  Signed-Rank  Test  was  the 
appropriate  test  to  utilize  in  analyzing  the  data.  If  the 
data  were  normally  distributed,  the  parametric  test  was 
used  (Step  2d).  If  the  data  were  not  normally  distributed, 
then  the  nonparametric  test  was  used  (Step  2c).  Typical 
examples  of  the  programs  used  to  perform  the  statistical 
analyses  are  provided  in  Appendices  C,  D,  and  E. 

Step  3.  Analyze  Test  Results 
* From  the  results  of  the  appropriate  statistical 

tests  (either  the  t-Test  or  the  Wilcoxon  Test)  conclusions 
can  be  made  regarding  the  direction  of  the  change  in  the 
mean  values  of  the  variables  being  analyzed. 
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Step  4.  Compare  Statistical  Conclusions  With  Criteria 

The  conclusions  developed  in  the  previous  step  re- 
garding the  modification's  effects  on  the  test  variables 
were  compared  to  the  established  criteria. 

Step  5.  Determine  Effectiveness  Of  Modification 

The  results  of  the  statistical  analyses  were  compared 
with  the  established  criteria.  The  results  of  this  compar- 
ison determined  the  conclusions  reached  regarding  the 
specific  effects  of  the  modification  and  determined  whether 
the  modification  was  effective. 

STATISTICAL  TESTS 

Three  statistical  tests  were  utilized  in  this  tech- 
nique. The  Kolmogorov-Smirnov  Goodness -of-Fit  Test  was  used 
to  determine  which  of  the  variables  being  studied  were 
normally  distributed.  If  normality  could  be  proven,  then 
the  Paired  t-Test  was  used  to  determine  if  there  was  any 
statistically  significant  difference  between  the  means 
before  and  after  the  modification  under  consideration.  If 
the  data  were  not  normally  distributed,  the  Wilcoxon 
Matched-Pairs  Signed-Ranks  Test  was  used  to  determine  if 
any  statistically  significant  difference  existed  between  the 
means.  The  following  discussion  describes  in  detail  the 
specific  statistical  tests. 
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Kolmogorov-Smirnov 
Goodness -of-Fit  Test 


In  this  test,  it  was  important  to  make  some  conclu- 
sions with  some  level  of  significance  about  the  nature  of 
the  distribution  of  the  population  from  which  the  sample 
data  were  drawn.  In  particular,  it  was  important  to  know 
which  of  the  variables  under  study  were  normally  distributed. 
By  knowing  this,  it  was  known  what  type  of  paired  difference 
test  to  use  on  the  data.  If  the  variables  were  considered 
not  to  be  normally  distributed  at  some  confidence  level, 
then  the  Wilcoxon  Matched-Pairs  Signed-Rank  Test  was  the 
appropriate  nonparametric  test  (23:19). 

The  Kilmogorov-Smirnov  Goodness -of-Fit  Test  was 
selected  over  the  Chi-Square  Goodness-of-Fit  Test  because 
the  Chi-Square  Test  is  not  the  most  efficient  test  to  fit 
sample  data  to  an  assumed  population  distribution.  In  the 
Chi-Square  Test  the  random  variables  must  be  grouped  into 
classes  and  the  numeric  value  of  each  individual  observa- 
tion is  lost  (21:681). 

Siegel  discussed  this  test  in  the  following  manner: 

The  Kilmogorov-Smirnov  one-sample  test  treats 
individual  observations  separately  and  thus,  un- 
like the  Chi-square  test  for  one  sample,  need  not 
lose  information  through  combining  of  categories. 

When  samples  are  small,  and  therefore  adjacent 
categories  must  be  combined  before  Chi-square  may 
properly  be  computed,  the  Chi-square  test  is 
definitely  less  powerful  than  the  Kilmogorov- 
Smirnov  test.  Moreover,  for  very  small  samples 
the  Chi-square  test  is  not  applicable  at  all,  but 
the  Kolmogorov-Smirnov  test  is.  These  facts  sug- 
gest that  the  Kolmogorov-Smirnov  test  may  in  all 
cases  be  more  powerful  than  its  alternative,  the 
Chi-square  test  £23:51]* 
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The  data  in  each  case  were  assumed  to  be  random, 
ratio  level,  and  continuous.  The  test  was  run  utilizing 
a program  specified  in  the  Northwestern  University  sup- 


plement to  SPSS  (26:12).  Continuity  was  assumed  even  though  | 

maintenance  actions  and  failures  have  discrete  values.  \ 

The  following  discussion  of  this  test  is  adapted 
from  Siegel's  text  (23:47-51).  The  test  involves  specify-  ' 

ing  the  cumulative  frequency  as  being  normally  distributed. 

The  cumulative  frequency  distribution  is  that  distribution 
which  is  felt  would  occur  under  the  theoretical  distribution 
and  is  compared  with  the  observed  cumulative  frequency  dis- 
tribution. The  theoretical  distribution  represents  what 
would  be  expected  under  the  null  hypothesis  (Hq).  Let  F(X) 
be  the  specified  cumulative  distribution  and  S(X)  the  ob- 
served. Under  the  null  hypothesis  that  the  sample  has  been 
drawn  from  the  specified  theoretical  distribution,  it  is 
expected  that  for  every  value  of  X,  S(X)  should  be  fairly 
close  to  F(X).  Under  Hq,  then,  we  would  expect  the  dif- 
ferences between  S(X)  and  F(X)  to  be  small  with  a 10  per 
cent  chance  of  random  error.  The  SPSS  program  computes 
the  value  at  which  the  greatest  divergence  occurs . The  ab- 
solute value  of  the  largest  value  is  called  the  maximum 
deviation,  D,  and  is  defined  as: 

Djjax  = Maximum  I F(X)  - S(X)  I 
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where: 

F(X)  is  the  specified  cumulative  distribution  and 
S(X)  is  the  observed  distribution. 


In  this  test  one  must  also  apply  the  Lilliefors 
modification  since  the  program  calculates  the  parameters 
for  the  theoretical  distribution  from  the  sample  (15) • The 
following  will  be  the  general  hypotheses  for  each  variable, 
the  test  statistic,  and  the  decision  rule: 

General  Statistical  Hypotheses: 

Hq  : The  random  sample  came  from  a normal  distri- 
bution with  mean  (X)  and  standard  deviation 
(S)  calculated  from  sample  values. 

Ha  : The  cumulative  distribution  function  is  not 
normal . 

Test  Statistic: 

l I X-  - X 

DMax  = I F(Z ) - S(Z)  I , where  Z = — , and 


1 — 1,2,.  . . n 
Decision  Rule: 

Reject  Hq  if  Dj^  is  greater 

.80' 


Dr  i + = - — * , where  1 - a = 
Crit  V n 


than  DCrit. 

•90  and  n is  sample 


size  ( 15 ) - 

This  test  was  run  on  34  data  sets  which  are  defined 
in  Appendix  A. 
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Paired  t-Test 


The  Paired  t-Test  was  utilized  to  determine  differ- 
ences in  population  means . The  test  assumes  a dependence 
between  the  populations  being  compared.  In  this  study,  it 
was  desired  to  compare  the  mean  values  for  flying  hours, 
sorties,  maintenance  actions,  maintenance  manhours, 

NORM,  NORS,  and  failures  before  modification  of  a group  of 
aircraft  with  the  mean  values  of  the  same  variables  and 
same  group  of  aircraft  after  the  modifications . 

The  Paired  t-Test  is  appropriate  in  many  experi- 
ments in  which  the  researcher  is  primarily  interested  in 
discovering  and  evaluating  differences  in  effects  rather 
than  the  effects  themselves.  In  this  study,  the  researchers 
were  concerned  with  how  a modification  changes  some  vari- 
ables related  to  aircraft  reliability,  maintainability,  and 
availability  rather  than  the  specific  values  that  relate  to 
reliability,  maintainability,  or  availability. 

Another  reason  for  desiring  to  use  the  Paired  t-Test 
in  this  study  was  to  eliminate  the  extraneous  influences  of 
material  handling,  environmental  effects,  maintenance  policy, 
mission,  and  facilities  on  the  variables  that  are  being 
analyzed.  In  the  Paired  t-Test,  only  the  paired  difference 
variable  D = Xg  - XA  is  formed,  where  Xg  is  the  measurement 
of  the  variable  before  the  modification  and  is  the  meas- 
urement after.  For  any  pair  of  variables  it  was  assumed 
that  the  extraneous  factors  were  the  same  before  as  after 
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the  modification.  The  difference,  then,  between  the  meas-  j 

urement  before  and  after  the  modification  was  an  estimate  1 

4 

of  the  difference  due  to  the  modification.  This  type  of  I 

statistical  analysis  leads  to  a more  accurate  comparison  i 

because  the  data  are  relatively  more  homogeneous  (20:2). 

To  be  able  to  utilize  the  Paired  t-Test,  the  sample 
data  have  to  be  normally  distributed.  The  results  of  the 
Kolmogorov-Smirnov  Test  was  used  to  determine  with  a 90  per 
cent  confidence  level  which  of  the  paired  sample  data  were 
normally  distributed.  The  data  were  discrete  and  ratio 
level. 

An  SPSS  program  was  used  to  accomplish  the  Paired 
t-Test  on  the  sample  modifications.  The  program  computes 
the  differences  in  the  sample  means  of  the  test  variables.* 

The  SPSS  program  computes  the  mean  difference  by  subtract- 
ing before  data  from  after  data.  The  following  test  sta- 
tistics, statistical  hypotheses,  and  decisions  rules  were 
used  for  this  test: 

Statistical  hypotheses  for  flying  hours  and  sorties: 

HQ  : FLY  A = FLY  B 

HA  : FLY  A / FLY  B 

Hq  « SOR  A » SOR  B 

Ha  i SOR  A / SOR  B 


*The  variables  represented  by  these  and  other 
abbreviations  are  included  in  Appendix  A. 
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Two-tailed  tests  were  accomplished  since  the  de- 
sired result  was  to  show  that  flying  hours  and  sorties  did 

<« 

not  change  in  the  data  set.  If  this  result  was  shown  to  be 

true  at  the  90  percent  confidence  level,  then  flying  hours  i 

and  sorties  were  shown  not  to  have  been  a factor  in  any 

significant  changes  in  the  other  five  paired  variable  tests. 

If  Hq  was  not  rejected,  it  was  concluded  at  a 90  percent 
confidence  level  that  aircraft  activity  was  the  same  on 
either  side  of  the  modification.  If  this  conclusion  was 
not  obtained,  then  the  results  of  the  remaining  variable 
tests  on  maintenance  actions,  maintenance  manhours,  NORM, 

NORS , and  failures  were  not  conclusive  as  to  what  effect 
the  modification  had  on  the  mean  value  of  the  variable 
tested. 

Statistical  hypotheses  for  other  variables: 

Hq  s MAB  < MAA 

Ha  5 MAB  > MAA 

Hq  i MHB  < MHA 

HA  j MHB  > MHA 

Hq  « NORM  B < NORM  A 
Ha  j NORM  B > NORM  A 
H0  s NORS  B < NORS  A 
Ha  i NORS  B > NORS  A 
Hq  « FAIL  B < FAIL  A 
Ha  i FAIL  B > FAIL  A 
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If  the  conclusion  was  made  that  flying  hours  and 
sorties  were  significantly  different  then  the  effects  of 
this  difference  were  eliminated  by  normalizing  the  var- 
iables. This  was  accomplished  by  dividing  maintenance 
actions,  maintenance  manhours,  NORM,  NORS,  and  failures  by 
their  respective  flying  hours  and  sorties.  The  following 
are  the  statistical  hypotheses  for  the  resulting  variables: 


H0 

: 
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< 
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FAILBS 

> 
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One-tailed  tests  were  utilized  since  the  analysts 
desired  to  conclude  that  the  variables  decreased  after  the 
modification  with  a 90  percent  confidence  level  when  com- 
pared to  the  test  variables  before  the  modification. 

Test  statistic: 

+ 1 ~ ^d 

s " v vr 
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where  d is  the  average  of  the  differences,  is  the 
difference  between  means  (before  minus  after),  sd  is  the 
standard  deviation  of  the  differences,  and  n is  the  sample 
size,  as  defined  in  Pfaffenberger  and  Patterson  (21:353). 

For  each  modification,  this  test  statistic  was  computed  by 
a SPSS  program  for  each  of  the  paired  hypothesis  tests. 

Decision  rules: 

1.  Reject  Hq  if  | tg | (absolute  value)  is  greater 
than  1.64  (two-tailed  test,  a = .10,  n - 1 > 30).  This 
decision  rule  was  used  for  the  hypotheses  for  flying  hours 
and  sorties. 

2.  Reject  Hq  if  tg  is  greater  than  1.28  (one-tailed 
test,  a = .10,  n - 1 > 30).  This  decision  rule  was  used  for 
the  hypotheses  for  the  unnormalized  and  normalized  mainte- 
nance action,  maintenance  manhours,  NORM,  NORS,  and  failure 
variables . 

Wilcoxon  Matched-Pairs 
Signed-Ranks  Test 

When  the  assumption  of  normality  is  not  reasonable 
for  the  data  being  analyzed,  nonparametric  tests  should  be 
utilized  to  test  hypotheses.  The  Wilcoxon  Matched-Pairs 
Signed-Ranks  Test  is  the  nonparametric  counterpart  to  the 
parametric  Paired  t-Test  for  testing  differences  between 
two  moans.  This  test  was  appropriate  for  those  variables 
in  the  study  that  were  shown  not  to  fit  the  theoretical 
normal  distribution  in  the  Kolmogorov-Smimov  Test  at  an 
a value  of  .10. 
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These  non-normal  sample  populations  were  tested 


with  the  Wilcoxon  Test  to  evaluate  what  effects  a modifica- 
tion had  on  the  normalized  and  not  normalized  means  of  the 
maintenance  action,  manhour,  NORM,  NORS , and  failure  var- 
iables before  the  modification  and  after  the  modification. 
This  Wilcoxon  Test  retains  the  convenience  of  the  Paired 
t-Test  in  that  it  minimizes  the  effects  of  such  factors , 
for  example,  as  maintenance  policy  and  geographical  location 
on  the  data.  The  Wilcoxon  Test  is  a more  powerful  test  than 
the  nonparametric  sign  test  in  that  it  not  only  provides 
information  about  the  direction  of  difference  within  the 
paired  data,  but  it  also  indicates  the  magnitude  of  the 
difference  (23:75). 

The  Northwestern  Supplement  to  SPSS  described  the 
Wilcoxon  Test  and  provided  the  program  that  analyzed  the 
data  (26:21).  The  explanation  of  the  results  in  this  sup- 
plement are  somewhat  misleading,  however  , in  that  it  did 
not  provide  a means  for  determining  the  variable  that  the 
test  was  computed  for.  Manual  calculations  were  required 
to  determine  this  information.  This  deficiency  makes  the 
program  somewhat  inconvenient. 

The  following  discussion  of  the  test  is  condensed 
from  Siegel  (23:75-83)*  The  Wilcoxon  Test  first  computes 
the  difference  (d)  in  the  means  between  each  set  of  matched 
pairs.  Each  pair  has  one  (d).  Each  d is  then  ranked 
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without  regard  to  sign.  The  rank  of  1 is  given  to  the 
smallest  d,  2 to  the  next  smallest,  and  so  on  until  all  the 
d's  are  ranked.  If  there  is  no  difference  in  the  pair,  then 
these  sets  of  data  are  discarded  and  ignored  in  the  anal- 
ysis. If  there  are  different  sets  of  data  with  the  same  d 
then  such  tied  cases  are  assigned  the  same  rank.  The  rank 
assigned  is  the  average  of  the  ranks  which  would  have  been 
assigned  if  the  d's  had  differed  slightly.  As  Siegel  dis- 
cussed, the  practice  of  giving  tied  observations  the  average 
of  the  ranks  they  would  otherwise  have  been  assigned  has  a 
negligible  effect  on  Tg  which  is  the  test  statistic  on  which 
the  Wilcoxon  Test  is  based  and  which  is  defined  below  (23:77). 

The  next  step  in  the  Wilcoxon  Test  assigns  the  sign 
of  the  difference  to  each  rank.  This  step  indicates  which 
ranks  arose  from  negative  d's  and  which  ranks  arose  from 
positive  d's.  The  sum  of  the  negative  ranks  and  the  sum  of 
the  positive  ranks  are  computed.  The  value  of  Tg  is  as- 
signed to  the  smaller  sum  of  like-signed  ranks.  The  value 
of  N (the  number  of  matched  pairs  minus  the  number  of  pairs 
whose  d = 0)  is  then  computed. 

Statistical  Hypotheses: 

The  same  statistical  hypotheses  were  used  in  this 
test  as  in  the  Paired  t-Test.  The  following  test  statistic 
and  decision  rule  will  apply  to  the  hypotheses: 
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Test  Statistic! 

Tg  = Smaller  value  of  n^X  (-Ranks  Mean)  and 


where , 


n^X  (+ Ranks  Mean) 


N ' nA  * nB 


and  where  nA  is  associated  with  after-modification  data  and 
nB  is  associated  with  before -modification  data.  SPSS  com- 
putes the  difference  in  this  study  as, 

Difference  = After  - Before 


Decision  rule; 

Reject  H0  if  Zs  > ZCrit  for  a = .10 


If  N > 25,  then  the  following  test  statistic  and 
decision  rule  will  be  used  to  test  the  hypotheses: 


and  Ts  is  as  defined  above  (20:79). 

If  N < 25  then  the  Wilcoxon  Test  is  not  normally 
utilized  and  appropriate  statistical  tables  are  used.  How- 
ever, Siegel  indicated  that  the  test  statistic  and  decision 
rules  for  N > 25  are  excellent  approximations  for  small 
samples,  N <25  (23:79).  Therefore,  in  this  study  the 
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Wilcoxon  Test  was  applied  in  those  few  cases  when  N < 25. 

For  two-tailed  tests  when  HQ  is  of  the  Form  X0  = XA: 

1.  If  Z0  > 1.64-  reject  HQ  at  90  percent  confidence. 

2.  If  Zg  < -1.64  reject  HQ  at  90  percent  confidence. 

For  one-tailed  tests  when  HQ  is  of  the  form  X0  < XA: 

1.  If  T0  is  associated  with  negative  differences, 
or  in  other  words,  if  n.x  (-mean  ranks)  < n_x  (+  mean  ranks) 

A -D 

then  if  Z0  > 1.28  then  reject  HQ  at  confidence  level  of 
90  percent. 

2.  If  T0  is  associated  with  positive  differences 

or,  in  other  words,  if  n^x  (-mean  ranks)  > n0x  (+  mean  ranks) 
then  if  Zs  < - 1.28  (or  Z0  is  a more  negative  value)  then 
reject  HQ  at  confidence  level  of  90  percent. 

DESIGN  TO  MEET  OBJECTIVES 

The  primary  objective  of  this  study  was  to  develop 
a technique  for  evaluating  the  effectiveness  of  Class  IV 
modifications  in  reducing  logistics  support  requirements. 

A secondary  objective  was  to  evaluate  selected  completed 
Class  IV  modifications  to  determine  if  they  were  effective 
in  reducing  logistics  support  requirements.  Some  of  the 
modifications  evaluated  had  reduction  of  logistics  support 
requirements  as  a secondary  purpose.  However,  the  evalua- 
tion of  effectiveness  related  only  to  effectiveness  in  re- 
ducing support  requirements.  As  a result,  a modification, 
designed  for  safety  or  mission  reasons,  may  have  been 


55 


effective  for  its  primary  purpose  but  be  judged  ineffective 
in  this  study  because  it  failed  to  achieve  a secondary 
purpose . 


Objective  1 

There  is  no  generally  accepted  method  for  deter- 
mining the  effectiveness  of  Class  IV  modifications  against 
which  this  technique  can  be  compared.  The  reader  is  en- 
couraged to  critically  evaluate  the  proposed  technique  and 
to  determine  the  efficacy  of  this  approach  for  his  purposes. 

Objective  2 

The  following  steps  were  followed  to  determine  the 
effectiveness  of  the  modifications  evaluated  during  the 
course  of  this  research: 

1.  A variable  or  variables  was  selected  as  repre- 
sentative of  the  logistics  support  element  which  the  AFLC 
Form  48  indicated  was  to  be  improved. 

2.  The  mean  value  of  the  selected  variable  before 
the  modification  was  statistically  compared  with  the  mean 
value  of  the  variable  after  the  modification. 

3.  The  modification  was  considered  effective  if  the 
results  of  the  statistical  hypothesis  test(s)  allowed  the 
strong  conclusion  that  the  mean  value  of  the  variable  se- 
lected in  Step  1 above  was  greater  before  the  modification 
than  after. 
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4.  The  modification  was  determined  to  be  ineffec- 


tive ifs 

a.  the  results  of  the  statistical  hypothesis 
test  indicated  that  the  mean  value  of  the  variable (s) 
representative  of  the  purpose  of  the  modification  was  not 
less  after  the  modification  than  before  or 

b.  the  results  of  the  statistical  hypothesis 
test(s)  indicated  that  the  mean  value  of  any  other  measured 
variable  was  greater  after  the  modification  than  before. 

ASSUMPTIONS 

1.  In  the  statistical  tests  utilized  in  this  study, 
a confidence  level  of  90  percent  was  used.  This  confidence 
level  was  utilized  since  it  is  the  confidence  factor  that  is 
used  in  current  military  specifications  dealing  with  relia- 
bility. As  Thomas  A.  Budne  explained  in  his  article  en- 
titled, "Basic  Philosophies  in  Reliability, " a 90  percent 
confidence  interval  is  utilized  to  insure  that  "there  are 
fewer  than  ten  percent  of  the  production  equipments  which 

do  not  meet  the  reliability  requirements  [ 5 » 91 D • ” 

2.  The  previously  described  factors  of  material 
handling,  environmental  conditions,  maintenance  policy, 
mission,  and  facilities  were  assumed  to  have  minimal  effect 
on  the  variables  analyzed.  The  assumption  was  made  that 
the  variables  analyzed  were  affected  by  these  factors  to 
the  same  degree  before  the  modification  as  after  the 
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modif ication.  Each  aircraft  served  as  its  own  control. 

This  assumption  made  it  easier  to  compare  the  differences 
between  the  measured  variables.  Any  differences  between 
the  variables  then  was  assumed  to  be  a result  of  the  par- 
ticular modification. 

3.  It  was  assumed  that  the  inconsistencies  that 
exist  in  the  data  as  a result  of  data  collection  and  docu- 
mentation errors  were  present  to  the  same  degree  in  the 
data  compiled  before  the  modifications  as  were  present  in 
the  data  compiled  after  the  modification.  By  comparing 
differences  in  the  variables  it  was  felt  that  these  data 
collection  errors  were  greatly  reduced  or  essentially 
eliminated. 

4.  It  was  assumed  that  a modification  affected 
only  that  piece  of  equipment  (WUC)  identified  on  the  AFLC 
Form  48.  This  study  did  not  attempt  to  determine  the 
effects  of  the  modification  on  other  equipment  or  systems 
on  the  aircraft. 

5.  It  was  assumed  that  all  work  accomplished  on 
the  modified  equipment  was  documented  by  the  technicians 
against  the  WUC  that  appeared  on  the  AFLC  Form  48.  Attempts 
to  regain  data  coded  to  higher  or  lower  level  WUC's  were  not 
made . 

6.  It  was  assumed  that  the  effects  of  exchanging 

modified  equipment  between  aircraft  (by  cannibalization  or 

other  maintenance  actions)  was  not  significant  enough  to 

affect  the  statistical  analyses  used  in  this  study. 
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SUMMARY 


\ 

This  chapter  discussed  the  specific  technique  that 
was  developed  to  analyze  the  effectiveness  of  a modifies-  ^ 

tion.  A flow  chart  was  constructed  to  illustrate  the  steps 
used  in  the  analyses  of  data.  The  specific  statistical 
tests  were  explained  in  detail  to  emphasize  the  components 
of  each  test  and  the  rationale  utilized  in  developing  the 

conclusions.  Finally,  the  criteria  used  to  measure  the  ■ 

effectiveness  of  a modification  and  the  assumptions  made  in 

this  study  were  explained.  < 
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Chapter  4 


ANALYSIS  OF  DATA 

This  chapter  provides  a detailed  discussion  of  the 
analyses  performed  on  fourteen  modifications  against  which 
the  technique  developed  in  Chapter  3 was  applied.  Complete 
displays  of  the  analysis  results  of  Modification  Number 
F65087B  (Case  9)  are  provided.  The  results  of  the  analyses 
for  the  remaining  13  modifications  are  summarized.  Case  9 
was  selected  for  detailed  presentation  primarily  because  it 
appeared  to  be  a "worst  case"  and  provided  a basis  for  dis- 
cussion of  all  aspects  of  the  analysis  technique.  Descrip- 
tive information  for  the  modifications  evaluated  is  pro- 
vided in  Table  4-1 . 


GENERAL  COMMENTS 

The  following  comments  apply  to  all  analyses  per- 
formed. 

Normality  of  Variables 

The  results  of  the  Kolmogorov-Smirnov  Goodness-of- 
Fit  Test  allowed  the  use  of  the  parametric  t-Test  as  the 
primary  test  for  differences  in  only  seven  (7)  of  208 
possible  instances.  This  occurred  because  in  only  these 
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MODIFICATIONS  EVALUATED 


6 


•Secondary  Purpose 


seven  instances  was  it  found  that  both  the  before  and  after 
data  were  normally  distributed.  There  were  12  other  in- 
stances of  normally  distributed  data.  However,  in  these 
instances  the  data  were  distributed  normally  on  only  one 
side  of  the  modification.  It  was  therefore  determined  that 
the  Wilcoxon  Matched-Pairs  Signed-Ranks  Test  would  be  the 
primary  method  used  to  test  for  decreases  in  the  mean  value 
of  a variable  after  a modification.  However,  the  t-Test 
was  also  applied.  If  the  t-Test  conclusions  differed  from 
those  of  the  Wilcoxon  Test,  the  Wilcoxon  Test  conclusions 
were  the  ones  accepted  unless  this  occurred  on  one  of  the 
seven  instances  when  the  t-Test  applied  directly.  On  the 
other  hand,  support  of  the  Wilcoxon  Test  conclusions  by  the 
t-Test  strengthened  the  Wilcoxon  conclusion. 

Normalized  Variable  Data 

Statistical  tests  for  equality  of  the  activity  in- 
dicators (flying  hours  and  sorties)  before  and  after  a 
modification  showed  that  in  most  cases  (19  of  28  cases) 
activity  was  significantly  different.  It  was  therefore 
concluded  that  no  useful  information  would  be  provided  by 
inclusion  of  the  analysis  of  nonnormalized  variables. 
Sometimes  analysis  of  normalized  variables  provided  dif- 
ferent conclusions  according  to  whether  the  variable  data 
was  normalized  by  flying  hours  or  by  sorties.  In  these 
cases,  it  was  necessary  for  the  researchers  to  determine 
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( judgment ally ) , on  a case-by-case  basis,  which  of  the  two 
activity  indicators  (flying  hours  or  sorties)  was  more 
appropriate . 

Application  of  Criteria 

The  criteria  for  determining  the  effectiveness  of 
a modification  were  as  follows: 

1 . The  mean  value  of  the  variable  representative 
of  the  purpose  of  the  modification  indicated  on  the  AFLC 
Form  48  was  greater  before  the  modification  than  after. 

2.  The  mean  values  of  other  measured  variables  were 
not  greater  after  the  modification  than  before. 

When  the  results  of  the  statistical  test  on  the  ap- 
propriate hypotheses  allowed  the  strong  conclusion  that  the 
first  criteria  listed  above  had  been  satisfied,  further 
analyses  were  required  to  determine  if  the  second  criteria 
was  also  satisfied.  These  analyses  were  accomplished  by 
testing  each  variable  whose  mean  value  appeared  to  remain 
the  same  or  increase  after  modification  with  a hypothesis 
test  of  the  form: 

Hq  : Variable  Mean  Value  Before  > Variable  Mean  Value  After 
Ha  : Variable  Mean  Value  Before  < Variable  Mean  Value  After 

If  the  null  hypothesis  (HQ)  was  not  rejected,  there 
was  no  statistically  significant  difference  between  the  mean 
value  of  the  variable  before  and  after  the  modification. 
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Otherwise,  the  mean  value  of  the  variable  was  greater  after 
the  modification. 


ANALYSIS  OF  CASE  9 

Step  1,  Criteria 

This  modification  "Cargo  Ramp  Overcenter  Lock  Condi- 
tion" was  a Class  IVB  modification  designed  to  prevent  fail- 
ure of  the  cargo  ramp  lock  hooks  and  attendant  cabin  depres- 
surization. The  AFLC  Form  48  description  of  the  modification 
also  stated  that  incorporation  of  the  modification  would 
"reduce  extensive  unprogrammed  manhours/aircraf t downtime 
required  for  a continuous  ramp  rigging  inspection.  ..." 

The  criteria  selected  for  evaluation  of  the  effectiveness 
therefore  became: 

1 . The  mean  value  of  manhours  expended  against  the 
work  unit  code  modified  (WUC  11BAC)  must  be  less  after  the 
modification  than  before. 

2.  The  mean  value  of  NORM  hours  charged  against 
WUC  11BAC  after  the  modification  must  be  less  than  those 
charged  against  WUC  11BAC  before  the  modification. 

3.  The  mean  value  of  other  measured  variables, 
i.e.,  maintenance  actions,  NORS,  and  failures,  must  be  no 
greater  after  the  modification  than  before. 

It  was  determined  that  sorties,  rather  than  flying 
hours,  would  be  the  appropriate  measure  of  cargo  ramp 
activity  because  the  cargo  ramp  is  opened  and  closed 
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in-flight  only  on  specialized  missions.  It  was  also  felt 
that  this  system  would  he  affected  more  by  landings  than 
by  flying  hours  accumulated. 

Step  2a.  Determination  of  Data  Distribution  for  Case  9 
The  Kolmogorov-Smimov  Goodness-of-Fit  Test  was 
performed  at  the  90  percent  confidence  level  on  the  34 
variables  associated  with  Case  9 to  determine  if  the  data 
were  normally  distributed.  In  only  one  instance  (Sorties 
After)  could  the  conclusion  be  reached  that  the  data  were 
normally  distributed.  In  the  other  33  instances,  the 
strong  conclusion  was  reached  that  the  data  were  not 
normally  distributed.  The  results  of  this  test  are  sum- 
marized in  Table  4-2. 

Step  2b.  Selection  of  Appropriate  Difference  Test 

From  the  results  of  the  Kolmogorov-Smimov  Test 
described  above,  it  was  determined  that  the  nonparametric 
Wilcoxon  Matched-Pairs  Signed-Ranks  Test  was  the  appropriate 
difference  test  to  apply  against  these  data. 

Step  2c.  Nonparametric  Test  for  Case  9 

This  Wilcoxon  Test  was  applied  against  Case  9 data 
normalized  by  both  sorties  and  flying  hours.  Sorties  were 
considered  to  be  the  appropriate  measure  of  system  activity. 
The  conclusions  reached  were  therefore  based  upon  the  re- 
sults of  tests  on  data  normalized  by  sorties. 
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1.  Data  normalized  by  sorties.  The  results  of  the  Wilcoxon 

statistical  hypothesis  tests  permitted  the  strong  conclu- 
sions to  he  drawn  that  the  mean  values  of  manhours  and  NORM  | 

hours  were  greater  before  the  modification  than  after  the  I 

modification  had  been  accomplished.  In  addition,  the  re- 
searchers were  able  to  conclude  that  the  mean  value  of 

neither  maintenance  actions,  NORS  hours,  nor  failures  had 
increased  after  the  modification.  The  results  of  the  ini- 
tial hypothesis  tests  indicated  that  the  mean  value  of 

- 

maintenance  actions  had  either  increased  or  remained  the  same 
after  the  modification.  It  was  therefore  necessary  to  test 
an  hypothesis  of  the  form  described  in  the  "Application  of 
Criteria”  section  of  this  chapter.  The  hypothesis  test 
used  was; 

Hq  : MABS  > MAAS 

Ha  ; MABS  < MR AS 

When  the  null  hypothesis  (HQ)  was  tested  at  the 
90  percent  confidence  level,  it  was  not  rejected.  It  was 
therefore  concluded  that  maintenance  actions  after  the  modi- 
fication (MAAS)  were  essentially  equal  to  maintenance  actions 
before  the  modification  (MABS). 

2.  Data  normalized  by  flying  hours.  Although  sorties  were 
considered  the  appropriate  activity  by  which  the  variables 
should  be  normalized,  analyses  of  the  variables  normalized 
by  flying  hours  were  also  accomplished.  The  Wilcoxon  Test 


67 


results  were  the  same  as  those  obtained  from  the  analyses 
of  variables  normalized  by  sorties  except: 

a.  Maintenance  actions  increased  after  the  modi- 
fication. 

b.  Manhours  remained  essentially  the  same  after  the 
modification  as  before. 

Step  2d.  Parametric  Test 

The  t-Test  was  also  performed  because  even  though 
the  data  were  not  normally  distributed,  corroboration  of 
the  results  of  the  nonparametric  test  by  the  parametric 
t-Test  provides  additional  strength  to  the  nonparametric 
test  results.  The  conclusions  reached  by  the  Wilcoxon  Test 
that  manhours  and  NORM  hours  were  less  after  the  modifica- 
tion than  before,  were  both  supported  by  the  t-Test  results. 
Also,  the  conclusion  that  maintenance  actions  were  essen- 
tially equal  before  and  after  the  modification  was  also 
supported.  However,  the  Wilcoxon  Test  conclusion  that  NORS 
hours  were  greater  before  the  modification  than  after  was 
not  supported  by  the  findings  of  the  t-Test  that  there  was 
no  statistically  significant  difference  between  the  NORS 
hours  before  and  after  the  modification. 

The  t-Test  results  for  variables  normalized  by  fly- 
ing hours  were  the  same  as  the  results  obtained  from  the 
analysis  of  variables  normalized  by  sorties  except: 

1 . Maintenance  actions  increased  after  the 
modification. 
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2.  There  was  no  statistically  significant  dif- 
ference between  failures  before  and  after  the  modification. 

The  results  of  the  Wilcoxon  Matched-Pairs  Signed- 
Ranks  Test,  the  t-Test,  and  a comparison  of  the  results  of 
these  tests  are  provided  in  Tables  4-3,  4-4  and  4-5 
respectively. 

Step  3.  Analyze  Test  Results 

For  this  modification,  it  was  determined  that  the 
number  of  sorties  flown  was  a better  measure  of  the  system 
activity  than  flying  hours.  Therefore,  the  conclusions 
reached  with  the  Wilcoxon  Test  using  variable  values 
normalized  by  sorties  were  the  ones  accepted.  These  con- 
clusions were: 

a.  The  mean  value  of  maintenance  hours  before  the 
modification  was  greater  than  the  mean  value  after  the 
modification. 

b.  The  mean  value  of  NORM  hours  before  the  modi- 
fication was  greater  than  the  mean  value  after  the  modi- 
fication. 

c.  The  mean  value  of  no  measured  variable  was 
greater  after  the  modification  than  before. 

These  conclusions  were  also  supported  by  the  con- 
clusions reached  using  the  t-Test  on  data  normalized  by 
sorties . 
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Table  4-5 
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Step  4.  Comparison  of  Statistical  Conclusions  with  Criteria 

The  analysis  of  Case  9 showed  that  when  normalized 
by  sorties  maintenance  hours  and  NORM  hours  were  less 
(statistically  significant)  after  the  modification  than 
before.  In  addition,  it  was  concluded  that  there  were  no 
statistically  significant  increases  in  the  mean  value  of 
other  measured  variables  when  normalized  by  sorties . In 
fact,  it  was  concluded  that  NORS  hours  and  failures  were 
also  less  (statistically  significant)  after  the  modifica- 
tion than  before.  Furthermore,  there  was  shown  to  be  no 
statistically  significant  difference  between  maintenance 
actions  before  and  after  the  modification. 

Step  5.  Effectiveness  Conclusion 

Modification  F65D87B  (Case  9)  was  considered  to  be 
effective  in  providing  improved  logistics  support. 

ANALYSIS  OF  THIRTEEN  OTHER  CASES 

The  following  discussion  explains  in  an  abbreviated 
manner  the  application  of  the  technique  to  thirteen  other 
modifications.  These  modifications  were  selected  to  demon- 
strate the  use  of  the  technique  on  what  is  assumed  to  be  a 
representative  sample  of  typical  Class  IV  modifications. 

As  explained  earlier,  the  first  five  modifications  do  not 
include  data  on  availability  (i.e.,  NORM  or  NORS),  and  have 
only  six  months  of  data  accumulated  as  opposed  to  18  months 
for  the  other  nine  cases. 
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Step  1.  Criteria 

The  criteria  utilized  in  evaluating  the  effective- 
ness of  these  13  modifications  are  identical  to  those  used 
in  evaluating  Case  9.  The  AFLC  Forms  48  were  not  available 
for  review  by  the  researchers  on  Cases  1-6.  These  cases 
were  initially  selected  by  other  individuals.  In  the  first 
five  cases,  the  purpose  of  the  specific  modification  was 
obtained  from  a draft  copy  of  Mitchell  and  Carter’s  study 
(20)  of  the  modifications.  The  purpose  of  Case  6 <vas  ob- 
tained from  personnel  assigned  at  San  Antonio  ALC.  In  the 
other  cases,  the  researchers  had  obtained  copies  of  the 
original  AFLC  Form  48  to  determine  the  modifications’ 
purpose . 

Step  2a.  Determination  of  Data  Distribution 

The  K-S  program  was  used  to  evaluate  the  data  to 
determine  which  sets  of  data  were  normally  distributed.  In 
all  thirteen  cases  the  only  data  that  were  normally  distrib- 
uted were  those  associated  with  equipment  activity,  i.e., 
flying  hours  and  sorties.  None  of  the  variables  associated 
with  equipment  reliability,  maintainability , or  availabil- 
ity followed  a normal  distribution. 

Step  2b.  Selection  of  Appropriate  Difference  Test 

From  the  results  of  the  K-S  test  the  parametric 
paired  t-Test  could  only  be  used  by  itself  in  seven  in- 
stances to  determine  the  direction  of  change  in  the  mean 
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value  of  the  variable  being  analyzed  at  a 90  percent  con-  ] 

fidence  level.  In  the  other  129  instances,  the  nonpara- 
metric  Wilcoxon  Matched-Pairs  Signed-Ranks  Test  became  the 

primary  method  for  testing  the  statistical  hypotheses.  < 

Step  2c.  Nonparametric  Test 

Since  the  majority  of  the  data  were  not  normally 
distributed,  the  nonparametric  Wilcoxon  Test  became  the 
appropriate  method  to  test  our  hypotheses  in  129  out  of  136 
instances.  The  hypotheses  accepted  from  these  tests  are 
shown  in  Table  4-6.  The  relationship  or  direction  of  change 
between  the  paired  variables  tested  (before  compared  to 
after)  is  shown  in  the  table.  If  the  final  hypothesis 
accepted  is  the  same  as  that  indicated  in  the  left  hand 
column,  then  a ”YES"  appears  in  the  appropriate  location 
in  the  table . 

Step  2d.  Parametric  Test 

Every  set  of  data  for  these  13  modifications  were 
analyzed  with  the  SPSS  t-Test  program.  In  only  one  in- 
stance out  of  136  tests  did  the  t-Test  not  support  the 
statistical  hypotheses  concluded  from  the  nonparametric 
Wilcoxon  Test.  This  difference  occurred  in  Case  7 where 
the  t-Test  indicated  at  a = .10  that  NORSBS  < NORSAS. 

The  Wilcoxon  Test  proved  the  alternate  hypothesis  at 
a = .10  that  NORSBS  > NORSAS.  However,  because  the  data 
for  NORSBS  and  NORSAS  were  not  normally  distributed,  it  had 
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Table  4-6 
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to  be  concluded  that  NORSBS  > NORSAS  since  the  parametric 
t-Test  could  not  be  employed  in  this  paired  analysis.  In 
all  the  other  135  conclusions  the  nonparametric  Wilcoxon 
Test  was  supported  by  the  parametric  t-Test  which  provided 
additional  strength  to  the  nonparametric  test  results. 

Step  3.  Analyze  Test  Results 

The  conclusions  from  the  appropriate  differences 
test  were  used  to  determine  the  changes  that  occurred  in 
the  test  variables.  Some  additional  analyses  needed  to  be 
performed  in  some  cases  to  determine  the  effect  of  the  modi- 
fication. For  example,  in  Case  4 maintenance  actions  and 
manhours  normalized  by  sorties  were  shown  by  the  Wilcoxon 
Test  to  have  decreased  after  the  modification.  However,  the 
null  hypotheses  (before  < after)  were  not  rejected  for  these 
two  variables  when  the  data  were  normalized  by  flying  hours. 
Because  of  this  difference  in  statistical  conclusions,  it 
became  necessary  to  determine  whether  the  variables  MA.AF 
and  MHAF  actually  had  been  increased  by  the  modification 
or  had  remained  unchanged.  It  was  determined  at  a 90 
percent  confidence  level  that  the  mean  value  of  these  two 
variables  (maintenance  actions  and  manhours)  had  not  in- 
creased after  the  modification. 

A similar  test  had  to  be  performed  for  Case  7.  In 
Case  7 the  following  conclusions  were  obtained  after  addi- 
tional testing: 
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The  results  of  this  comparison  are  shown  in  Table  4-7. 
The  variables  selected  for  comparison  are  based  on  the  stated 
purpose  as  indicated  on  the  AFLC  Form  48  and  the  relation- 
ships previously  established  in  this  study. 

Step  5.  Effectiveness  Conclusion 

In  these  thirteen  cases,  four  modifications  were  de- 
termined to  be  effective.  The  effectiveness  of  the  modifica- 
tions is  shown  in  Table  4-7 . 

It  should  be  noted  that  any  variables  that  were 
decreased  after  the  modification  but  were  not  included  as 
part  of  the  stated  purpose  of  the  modification  were  con- 
sidered irrelevant  in  determining  the  effectiveness  of 
the  modification.  This  occurred  in  Case  12  when  NORM  time 
(normalized  by  sorties  and  flying  hours)  was  shown  to  de- 
crease after  the  modification.  The  purposes  as  stated  on 
the  AFLC  Form  48  of  the  modification  were  to  prevent  fail- 
ures and  "more  expensive  repair  actions.”  The  analyses 
showed  that  maintenance  actions  and  failures  had  not  de- 
creased after  the  modification.  Therefore,  the  modification 
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SUMMARY  OF  MODIFICATION  EFFECTIVENESS 
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•Secondary  Purpose 


did  not  accomplish  its  intended  purpose  and  was  considered 
not  effective.  The  change  in  NORM  time  had  no  effect  on 
this  conclusion  and  was  not  considered. 

RESULTS  OF  ANALYSIS  OF  DATA 

Of  the  fourteen  modifications  selected,  five  were 
determined  to  be  effective  when  evaluated  by  the  developed 
technique.  The  following  is  a summary  of  the  effective 
modifications : 

Table  4-8 

SUMMARY  OF  EFFECTIVE  MODIFICATIONS 
CASE  NO.  TYPE 

2 IVA  Reduction  in  all  measured  variables* 

4 IVB  Reduction  in  MAS,  MHS , FAILS,  & FAILF* 

9 IVB  Reduction  in  MHS,  NORMS,  NORMF,  NORSS , 

NORSF,  NORMS,  NORMF,  FAILS,  FAILF 

10  IVB  Reduction  in  MAS,  MAF,  MHS,  MHF, 

FAILS,  & FAILF 

11  I VC  Reduction  in  all  measured  variables 

•NORM  & NORS  variables  not  measured. 

It  should  be  noted  that  only  one  out  of  the  three 
Class  IVC  modifications  selected  for  evaluation  actually 
achi*  ved  their  stated  purpose  of  reducing  logistic  support. 

The  specific  data  analyzed  and  the  computer  print- 
outs for  this  study  are  available  for  review  at  AFALD/XRSC, 
Wright-Pattersor.  AFB.  Typical  examples  of  the  K-S, 
Wilcoxon,  and  t-Test  programs  used  are  provided  in 
Appendices  C,  D,  and  E,  respectively. 
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SUMMARY 


This  chapter  provided  a detailed  discussion  of  the 
analyses  completed  on  Case  9 using  the  technique  developed 
in  Chapter  3.  Each  step  in  the  proposed  technique  was  ex- 
plained using  the  data  analyses  accomplished  in  this  case. 
The  results  of  the  tests  performed  on  the  other  13  cases 
were  also  described.  Finally,  a summary  was  provided  of 
those  modifications  that  were  shown  to  be  effective  after 
the  testing  had  been  completed. 
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Chapter  5 


SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

In  this  chapter  the  initial  discussion  will  pro- 
vide a summary  of  the  research  performed.  This  will  be 
followed  by  a presentation  of  the  resulting  conclusions 
of  the  research.  The  final  discussions  will  describe  some 
recommendations  for  additional  research  effort. 

SUMMARY 

This  study  has  shown  that  for  many  years  the  Air 
Force  has  been  approving  Class  IV  modifications  to  reduce 
aircraft  logistic  support  requirements  and  improve  air- 
craft reliability,  maintainability,  and  availability. 

While  there  are  extensive  reviews  by  Configuration  Control 
Boards  prior  to  their  approval,  no  systematic  technique 
exists  for  assessment  of  the  modifications  after  implementa- 
tion to  evaluate  their  effectiveness  in  achieving  their 
stated  purposes. 

Using  data  from  the  Air  Force  G098  system,  an 
assessment  technique  was  developed  in  this  study  which  used 
parametric  and  nonparametric  statistical  mean  difference 
tests  to  evaluate  the  effectiveness  of  Class  IV 
modifications. 
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Fourteen  selected  modifications  were  evaluated  to 
demonstrate  how  the  G098  data  were  compiled  and  analyzed 
by  this  technique.  Included  in  this  evaluation  were  data 
on  maintenance  actions,  manhours,  NORM,  NORS,  and  failures 
before  and  after  the  modification.  These  data  sets  were 
adjusted  for  variations  in  flying  hours  and  sorties.  The 
data  were  then  analyzed  to  determine  if  there  were  any 
significant  improvements  as  a result  of  the  Class  IV  modi- 
fication. A modification  was  determined  to  be  effective 
if  these  improvements  were  part  of  the  original  stated 
purpose  of  the  modification  as  indicated  on  the  AFLC 
Form  48. 

The  following  research  question  as  previously 
stated,  was  answered  in  the  affirmative:  Can  a systematic 
technique  which  uses  available  systems  data  be  developed 
to  determine  whether  or  not  the  stated  purposes  appearing 
on  the  AFLC  Form  48  as  part  of  the  original  justification 
for  Class  IV  modifications  have  been  achieved? 


CONCLUSIONS 

Conclusion  1 

Data  are  available  in  several  Air  Force  data  systems 
which  can  be  used  as  the  basis  of  a technique  for  evaluating 
the  effectiveness  of  aircraft  modifications  designed  to  im- 
prove logistics  support.  The  Configuration  Control  Board 
Item  Records  (AFLC  Form  48)  can  be  used  as  the  source  of 
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data  concerning  the  purpose  of,  and  items  affected  by,  the 
modifications.  The  Standard  Configuration  Management  Sys- 
tem (D0^7)  and  the  Advanced  Configuration  Management  System 
(D057G)  can  provide  information  concerning  the  aircraft 
serial  numbers  affected  by  a modification  and  the  date  each 
individual  serial  number  was  modified.  The  Maintenance  Re- 
quirements Data  System  (G098)  can  be  used  as  the  source  of 
aircraft  maintenance,  inventory,  utilization,  and  status  in- 
formation by  aircraft  serial  number.  The  combination  of  the 
above  data  sources  provide  the  data  necessary  for  develop- 
ment of  a technique  for  assessing  the  impact  of  a modifica- 
tion on  logistics  support  of  an  aircraft. 


Conclusion  2 

The  parametric  t-Test  and  the  nonparametric  Wilcoxon 
Matched-Pairs  Signed-Ranks  Test  are  difference  tests  which 
provide  viable  methods  for  comparing  the  mean  values  of 
variables  before  and  after  a modification.  Computerized 
application  of  these  tests  to  appropriately  matched  data 
allow  conclusions  to  be  derived  regarding  increases/ 
decreases  in  the  mean  values  of  the  variables  represented 
by  the  data. 

Conclusion  3 

The  technique  developed  in  this  study  can  be  used, 
under  the  conditions  indicated,  for  evaluating  the  effec- 
tiveness of  Class  IV  modifications  in  reducing  aircraft 

logistics  support  requirements. 
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Conclusion  4 

It  was  not  possible  to  conclude  from  this  study 
which  measure  of  activity  (flying  hours  or  sorties)  was  a 
better  indicator  of  aircraft  system  activity.  Instead  it 
was  concluded  that  the  researcher  must  evaluate  the  activ- 
ity measure  to  be  used  on  a case-by-case  basis.  This 
evaluation  will  be  somewhat  subjective.  Additionally,  the 
degree  of  confidence  placed  on  the  conclusions  derived  will 
depend  to  some  extent  upon  the  experience  of  the  researcher 
and  his  familiarity  with  the  system  being  studied. 

RECOMMENDATIONS 

Recommendation  1 

Additional  effort  should  be  expended  to  apply  this 
analytical  technique  to  other  aircraft  modifications . Modi- 
fications on  other  mission,  design,  or  series  (MDS)  aircraft 
should  be  evaluated  by  this  technique . 

Recommendation  2 

The  technique  should  be  assessed  to  determine  its 
applicability  to  the  assessment  of  the  impact  of  changes, 
other  than  Class  IV  modification,  on  logistics  support. 
Examples  of  such  changes  which  might  be  assessed  using  this 
technique  are* 

1.  Class  I Modifications, 

2.  Class  V Modifications,  and 

3.  Phase  Inspections. 
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Recommendation  3 

Research  should  continue  to  be  conducted  on  efforts 
to  develop  a method  of  costing  improvements  resulting  from 
modifications.  This  would  allow  computation  of  dollar 
benefits  resulting  from  modifications. 
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ABBREVIATIONS 

FLYB Flying  hours  per  aircraft  during  the  period  preceding 

a modification. 

FLYA Flying  hours  per  aircraft  during  the  period  following 

a modification. 

SORB Sorties  per  aircraft  during  the  period  preceding  a 

modification. 


SORA Sorties  per  aircraft  during  the  period  following  a 

modification. 

MAB Maintenance  actions  per  aircraft  on  the  modified 

equipment  during  the  period  preceding  a modification. 

MAA Maintenance  actions  per  aircraft  on  the  modified 

equipment  during  the  period  following  a modification. 

MABF MAB  divided  by  FLYB. 

MAAF MAA  divided  by  FLYA. 

MABS MAB  divided  by  SORB. 

MAAS  — -MAA  divided  by  SORA. 

MHB Manhours  per  aircraft  expended  to  repair  the  modified 

equipment  during  the  period  preceding  a modification. 

MHA Manhours  per  aircraft  expended  to  repair  the  modified 

equipment  during  the  period  following  a modification. 

MHBF MHB  divided  by  FLYB. 

MHAF MHA  divided  by  FLYA. 

MHBS MHB  divided  by  SORB. 

MHAS MHA  divided  by  SORA. 

FAILB — Failures  per  aircraft  of  the  modified  equipment 
during  the  period  preceding  a modification. 
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FAILA Failures  per  aircraft  of  the  modified  equipment 

during  the  period  following  a modification. 

FAILBF FAILB  divided  by  FLYB. 

FAILAF FALLA  divided  by  FLYA. 

FAILBS— - FAILB  divided  by  SORB. 

FALLAS—  FAILA  divided  by  SORA. 

NORMB Not  Operationally  Ready  due  to  Maintenance  hours  per 

aircraft  on  the  modified  equipment  during  the  period 
preceding  a modification. 

NORMA Not  Operationally  Ready  due  to  Maintenance  hours  per 

aircraft  on  the  modified  equipment  during  the  period 
following  a modification. 

NORMBF-- -NORMB  divided  by  FLYB. 

NORMAF— NORMA  divided  by  FLYA. 

NORMBS-— NORMB  divided  by  SORB. 


NORMAS— -NORMA  divided  by  SORA. 

NORSB Not  Operationally  Ready  due  to  Supply  hours  per 

aircraft  on  the  modified  equipment  during  the  period 
preceding  a modification. 

NORSA Not  Operationally  Ready  due  to  Supply  hours  per 

aircraft  on  the  modified  equipment  during  the  period 
following  a modification. 

NORSBF— NORSB  divided  by  FLYB. 

NORSAF— -NORSA  divided  by  FLYA. 

NORSBS—  NORSB  divided  by  SORB. 

NORSAS— NORSB  divided  by  SORA. 
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GLOSSARY  OF  TERMS 


Availability’s  The  probability  that  a system  or  equipment, 
when  used  under  stated  conditions  in  an  actual 
operational  environment,  will  operate  satisfactorily 
when  called  upon  [3:5]. 

Logistic  Support:  Includes  such  items  as  test  and  support 
equipment,  supply  support  (spares  and  repair  parts), 
personnel  and  training,  technical  data,  facilities, 
transportation  and  handling,  and  maintenance  planning 
(3:8-9). 

Maintainability:  A characteristic  of  design  and  installation 

which  is  expressed  as  the  probability  that  an  item  will 
be  retained  in  or  restored  to  a specified  condition  with- 
in a given  period  of  time,  when  maintenance  is  performed 
in  accordance  with  prescribed  procedures  and  resources 
[3:10]. 

Reliability:  The  probability  that  a system  or  product  will 

give  satisfactory  performance  for  a specified  period  of 
time  when  used  under  stated  conditions  ....  Reliability 
can  be  expressed  in  terms  of  mean  time  between  failures 
( MTBF)  or  mean  life  [3:12]. 

Supportability : Considers  maintenance  time,  logistic  re- 

sources, and  maintenance  cost  [3:19]. 
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Appendix  C 


K-S  PROGRAM 


1 OS AMPLE  OF  PROGRAM  USED  FOR  K-S  TEST 

20TRE 1 4, CM1 20000, T20t 1020, STCSB.  T770253, ED I SON /COLEMAN, 56513 
30ATTACH , SPS S 7 , ID-AFIT . 

40SFSS7. 

50  7/8/9 
60RUN  NAME 
70VARIABLE  LIST 
80 

90INPUT  FORMAT 
LOON  OF  CASES 
1 IOCOMPUTE 
1 20COMPUTE 
1 30COMPUTE 
I40C0MPUTE 
150COMPUTE 
160COMPUTE 
I 70C0MPUTE 
180COMPUTE 
I90COMPUTE 
200COMPUTE 
2 IOCOMPUTE 
220COMPUTE 
230COMPUTE 
240COMPUTE 
250COMPUTE 
260COMPUTE 
270COMPUTE 
280COMPUTE 
290COMPUTE 
300COMPUTE 
3 1 ON PAR  TESTS 
320 
330 
340 
350 
360 
370 
380 
390 
400 
410 
420 
430 
440 

450OPTIONS 
460READ  INPUT  DATA 
470PINISH 


K-S  TEST  FOR  MOD  14 

MO , SN , FL YB , FL YA , SORB , SORA , MAB , MAA , MHB , MHA , 

NORMB , NORMA, NORSB , NORSA, FAILB , FAILA 
FIXED (F2. 0,F8 .0, 2F5 . 1 ,4F4.0,6F5. 1 , 2F4.0) 

335 

MABS-MAB/SORB 

MAAS-MAA/SORA 

MHBS-MHB/SORB 

MHAS-MHA/SORA 

NORMBS -NORMB /SORB 

NORMAS-NORMA/SORA 

NORSBS-NORSB /SORB 

NORSAS-NORSA/SORA 

FAILBS-FAILB/SORB 

FAILAS-FAI LA/SO RA 

MAB  F-MAB / FL YB 

MAAF-MAA/FLYA 

MHBF-MHB/FLYB 

MHAF-MHA/FLYA 

NORMBF-NORMB /FL YB 

NORMAF-NORMA/FLYA 

NORSBF-NORSB/FLYB 

NORSAF-NORSA/FLYA 

FAILB F-FAILB/FLYB 

FAILAF-FAI  LA/Ft.  YA 

K-S (NORMAL ) -FLYB/K-S (NORMAL ) -FLYA/K-S 
(NORMAL ) -SORB /K-S (NORMAL ) -SORA/K-S (NORMAL ) -MABF/ 
K-S (NORMAL )-MAAF /K-S (NORMAL )-MHBF /K-S (NORMAL )- 
MHAF/K-S (NORMAL )-NORMBF/K-S (NORMAL )-NORMAF/K-S 
(NORMAL) -NORSBF /K-S (NORMAL ) -NORSAF/K-S (NORMAL ) 
-FAILBF/K-S (NORMAL )-FAILAF/K-S (NORMAL )-MABS /K-S ( 
NORMAL ) -MAAS /K-S (NORMAL ) -MHBS /K-S (NORMAL ) -MHAS / 
K-S (NORMAL ) -NORMBS /K-S (NORMAL ) -NORMAS /K-S (NORMAL ) 
-NORSBS /K-S (NORMAL ) -NORSAS /K-S (NORMAL ) -FAILBS / 

K-S (NORMAL )-FAI LAS /K-S (NORMAL ) -MAB /K-S (NORMAL )- 
MAA/K-S (NORMAL ) -MHB /K-S (NORMAL ) -MHA /K-S (NORMAL ) - 
NORMB /K-S (NORMAL ) -NORMA/ K-S (NORMAL ) -NORSB / 

K-S (NORMAL ) -NORSA/K-S (NORMAL ) -FAILB /K-S (NORMAL ) 

-FAILA 

3 
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Appendix  D 


WILCOX ON  PROGRAM 


100SAMPLE  OF  PROGRAM  USED  FOR  WILCOXON  TEST 

110TRE12,CM1 20000, T20, 1020, STCSB.  T770253,EDISON/COLEMAN,56513 
1 20ATTACH , SPSS  7 , ID-AFIT . 

130SPSS7. 

140  7/8/9 

150RUN  NAME  WILCOXON  TEST  MOD 12 

160VARIABLE  LIST  MD,SN,FLYB,FLYA,SORB,SORA,MAB,MAA,MHB,MHA, 

170  NORMB , NORMA , NORSB , NORSA , FAILB , FAILA 

180INPUT  FORMAT  FIXED(F2.0,F8.0, 2F5. 1 ,4F4.0,6F5. 1 , 2F4.0) 

190N  OF  CASES  265 

200COMPUTE  MABS-MAB/SORB 

210COMPUTE  MAAS-MAA/SORA 

220COMPUTE  MHBS-MHB /SORB 

230COMPUTE  MHAS-MHA/SORA 

240C0MPUTE  NORMBS-NORMB/SORB 

250COMPUTE  NORMAS-NORMA/SORA 

260COMPUTE  NORSBS-NORSB/SORB 

270COMPUTE  NORSAS-NORSA/SORA 

280C0MPUTE  FAILBS-FAILB/SORB 

290COMPUTE  FAILAS-FAILA/SORA 

300COMPUTE  MABF-MAB/FLYB 

310COMPUTE  MAAF-MAA/FLYA 

320COMPUTE  MHBF-MHB/FLYB 

330COMPUTE  MHAF-MHA/FLYA 

340COMPUTE  NORMBF-NORMB/FLYB 

350COMPUTE  NORMAF-NORMA/FLYA 

360COMPUTE  NORSBF-NORSB/FLYB 

370C0MPUTE  NORSAF-NORSA/FLYA 

380COMPUTE  FAILB F-FAI LB /FLYB 

390COMPUTE  FAILAF-FAILA/FLYA 

400NPAR  TESTS  WILCOXON-FL YB , SORB ,MAB , MHB , NORMB , NORSB , 

4 l 0 FAILB ,MABS , MHBS , NORMBS , NORSBS , FAILBS , 

420  MABF , MHBF , NORMBF , NORSBF , FAILBF  WITH 

430  FL YA , SORA , MAA , MHA, NORMA , NORSA, FAILA , 

440  MAAS, MHAS, NORMAS, NORSAS.FAI LAS, MAAF, 

450  MHAF , NORMAF , NORSAF , F AILAF 

460OPTIONS  3 

470READ  INPUT  DATA 
480FINISH 
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Appendix  E 


t-TEST  PROGRAM 


$ IDENT  WP1149.AFIT/LSG  COLEMAN  EDISON  78B 

$ SELECT*SPSS/BIGSPSS 

RUN  NAME  PAR  T-TEST, NORMALIZED  BY  FLY  HRS.  & SORTIES-MOD12 

COMMENT  THIS  PROGRAM  PERFORMS  THE  PARAMETRIC  T-TEST 

DESCRIBED  IN  THE  STUDY 

VARIABLE  LIST  MO,SN,FLYB,FLYA,SORB,SORA,MAB,MAA,MHB,MHA, 

NORMB , NORMA , NORSB , NORSA , FAILB , FAILA 
INPUT  FORMAT  FIXED(F2.0,F8.0,2F5. 1.4F4.0.6F5. 1.2F4.0) 

INPUT  MEDIUM  CARD 
ALLOCATE  TRANS PACE-5000 

N OF  CASES  265 

COMMENT  THE  FOLLOWING  COMPUTATIONS  ARE  REQUIRED  TO 

REMOVE  THE  EFFECTS  OF  FLYING  HOURS  AND/OR 
SORTIES  PRIOR  TO  PERFORMING  THE  T-TEST 
COMPUTE  MABF-MAB/FLYB 

COMPUTE  MAAF-MAA/FLYA 

COMPUTE  MHBF-MHB/FLYB 

COMPUTE  MHAF-MHA/FLYA 

COMPUTE  NORMB F-NORMB/FLYB 

COMPUTE  NORMAF-NORMA/FLYA 

COMPUTE  NORSBF-NORSB/FLYB 

COMPUTE  NORSAF-NORSA/FLYA 

COMPUTE  FAILBF-FAILB/FLYB 

COMPUTE  FAILAF-FAILA/FLYA 

COMPUTE  MABS-MAB/SORB 

COMPUTE  MAAS-MAA/SORA 

COMPUTE  MHBS-MHB/SORB 

COMPUTE  MHAS-MHA/SORA 

COMPUTE  NORMBS-NORMB/SORB 

COMPUTE  NORMAS-NORMA/SORA 

COMPUTE  NORSBS-NORSB/SORB 

COMPUTE  NORSAS-NORSA/SORA 

COMPUTE  FAILBS-FAILB /SORB 

COMPUTE  FAILAS-FAILA/SORA 

COMMENT  THE  T-TEST  WAS  PERFORMED  ON  ALL  BEFORE  AND 

AFTER  DATA  PAIRS  SO  RESULTS  COULD  BE  COMPARED 
WITH  NON-PARAMETRIC  TEST  RESULTS 
T-TEST  PAIRS-FLYB , FL YA/SORB , SORA/MAB .MAA/MHB ,MHA/ 

NORMB , NORMA /NORSB , NORSA /FAILB .FAILA/ 
MABF,MAAF/MHBF,MHAF/NORMBF,NORMAF/NORSBF, 

NORSAF / FAILB  F , FAILAF/MABS , MAAS /MHBS , MHAS / 

NORSBS , NORSAS /FAILBS , FAILAS 
READ  INPUT  DATA 
$ S ELECTA  MOD 12 

FINISH 

$ ENDJOB 
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